Oct 30 23:58:06.927035 ip-10-0-136-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:58:07.327115 ip-10-0-136-236 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:07.327115 ip-10-0-136-236 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:58:07.327115 ip-10-0-136-236 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:07.327115 ip-10-0-136-236 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:58:07.331013 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.330930 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.331013 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331011 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.331013 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331015 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.331013 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331018 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331021 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331024 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331027 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331030 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331033 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331036 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331038 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331041 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331047 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331050 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331054 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331058 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331061 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331064 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331067 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331069 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331072 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331075 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.331142 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331077 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331080 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331082 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331085 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331088 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331090 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331093 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331095 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331098 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331100 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331103 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331105 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331108 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331110 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331114 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331117 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331120 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331122 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331125 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331127 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.331585 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331130 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331139 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331142 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331144 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331147 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331150 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331153 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331155 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331158 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331161 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331163 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331166 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331168 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331171 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331173 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331176 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331178 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331180 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331183 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331187 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.332161 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331192 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331194 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331197 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331203 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331223 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331227 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331230 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331233 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331235 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331238 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331240 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331243 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331246 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331249 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331251 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331254 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331257 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.332629 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331259 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331262 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331264 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331266 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331269 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331271 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331274 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331276 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331279 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331281 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331283 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331286 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331288 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331290 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331293 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331295 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331298 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331302 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331305 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331311 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.333158 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331313 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331315 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331318 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331320 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.331323 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332125 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332131 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332134 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332137 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332140 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332142 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332145 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332148 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332151 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332153 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332156 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332158 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332161 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332163 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332166 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.333624 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332170 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332173 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332175 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332178 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332181 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332183 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332186 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332189 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332191 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332194 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332197 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332200 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332202 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332205 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332207 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332210 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332212 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332215 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332217 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.334118 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332219 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332221 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332224 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332227 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332229 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332232 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332236 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332239 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332241 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332244 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332246 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332249 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332251 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332253 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332256 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332258 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332261 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332263 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332266 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.334573 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332268 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332270 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332276 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332281 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332284 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332287 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332289 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332292 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332294 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332297 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332299 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332302 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332304 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332307 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332309 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332318 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332321 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332323 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332326 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.335057 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332328 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332331 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332333 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332335 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332338 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332341 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332344 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332347 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332349 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332352 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332354 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332357 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332360 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332362 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332365 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332369 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332372 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332374 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332376 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.335519 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332379 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332383 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332387 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332389 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332392 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332394 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332397 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332399 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332401 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332404 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332412 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332414 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332417 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.332419 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333046 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333054 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333058 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333061 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333064 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.335989 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333067 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333069 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333072 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333075 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333077 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333080 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333083 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333085 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333088 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333091 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333094 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333097 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333099 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333101 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333104 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333107 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333109 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333112 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333114 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333117 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.336432 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333119 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333121 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333124 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333129 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333132 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333135 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333137 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333139 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333142 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333144 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333148 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333152 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333155 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333158 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333161 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333164 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333167 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333170 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333174 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333178 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.336946 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333181 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333183 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333186 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333189 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333191 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333193 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333196 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333198 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333201 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333203 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333206 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333208 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333211 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333213 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333215 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333218 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333221 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333223 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333226 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.337414 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333228 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333231 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333233 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333235 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333238 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333240 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333243 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333245 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333247 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333250 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333252 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333256 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333259 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333261 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333264 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333267 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333269 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333272 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333274 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.337906 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333277 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333279 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333282 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333284 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333287 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333289 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333292 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333295 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333297 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333300 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333302 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333305 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333308 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333310 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333313 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333315 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333318 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333320 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333323 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333325 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.338354 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333328 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333333 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.333337 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334142 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334153 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334159 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334165 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334169 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334173 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334178 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334182 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334186 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334189 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334192 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334195 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334199 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334202 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334204 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334208 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334210 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334213 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334219 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334222 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:58:07.338839 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334225 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334228 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334232 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334236 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334238 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334241 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334245 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334248 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334251 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334254 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334257 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334260 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334264 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334269 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334272 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334277 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334280 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334283 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334287 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334290 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334293 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334296 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334299 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334303 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:58:07.339357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334306 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334309 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334312 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334315 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334318 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334321 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334324 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334327 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334330 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334332 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334337 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334340 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334343 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334347 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334367 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334371 2579 flags.go:64] FLAG: --help="false" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334374 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334378 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334382 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334385 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334388 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334393 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334396 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:58:07.339967 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334400 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334403 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334406 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334409 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334412 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334414 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334417 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334420 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334423 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334426 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334429 2579 flags.go:64] FLAG: --lock-file="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334433 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334435 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334438 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334444 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334448 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334451 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334453 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334456 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334460 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334463 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334466 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334471 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334474 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334478 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 23:58:07.340567 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334482 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334485 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334487 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334490 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334493 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334497 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334500 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334513 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334516 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334519 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334523 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334525 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334528 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334531 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334534 2579 flags.go:64] FLAG: --port="10250" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334537 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334540 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d378a1458ba62f5e" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334543 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334546 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334549 2579 flags.go:64] FLAG: --register-node="true" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334552 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334557 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334559 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334562 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334565 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334569 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:58:07.341218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334572 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334575 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334579 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334582 2579 flags.go:64] FLAG: --runonce="false" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334585 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334589 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334592 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334595 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334597 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334601 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334604 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334608 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334611 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334615 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334618 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334621 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334624 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334627 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334630 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334636 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334639 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334642 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334647 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334649 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334652 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:58:07.341859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334655 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334658 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334661 2579 flags.go:64] FLAG: --v="2" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334666 2579 flags.go:64] FLAG: --version="false" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334670 2579 flags.go:64] FLAG: --vmodule="" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334674 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.334678 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334833 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334836 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334840 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334843 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334846 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334848 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334851 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334854 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334856 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334859 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334862 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334866 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334868 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334872 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.342441 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334875 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334878 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334880 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334883 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334885 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334888 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334890 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334893 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334895 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334898 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334901 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334903 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334906 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334908 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334910 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334913 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334915 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334917 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334920 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334922 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.343078 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334924 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334927 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334930 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334932 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334935 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334937 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334940 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334942 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334946 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334950 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334954 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334958 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334961 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334964 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334967 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334969 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334972 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334975 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334978 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.343833 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334981 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334984 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334987 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334990 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334992 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334995 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.334997 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335000 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335002 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335005 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335007 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335010 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335012 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335016 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335019 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335022 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335026 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335029 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335031 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.344335 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335034 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335037 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335039 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335043 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335046 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335049 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335052 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335054 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335057 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335060 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335062 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335065 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335067 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335070 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335073 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335075 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335077 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335080 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335082 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335084 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.344848 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335087 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335089 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335093 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335095 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335098 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335100 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335103 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335105 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335107 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335110 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335113 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335115 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335118 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335120 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.335147 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335183 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.345310 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335189 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335193 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335196 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335198 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335201 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335203 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335206 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335209 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335211 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335214 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335216 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335219 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335221 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335224 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335227 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335229 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335232 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335234 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335237 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.345693 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335239 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335241 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335244 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335247 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335249 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335251 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335255 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335258 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335261 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335263 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335266 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335268 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335271 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335273 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335277 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335280 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335282 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335285 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335287 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335289 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.346155 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335292 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335294 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335297 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335299 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335302 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335304 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335306 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335310 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335314 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335316 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335319 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335322 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335324 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335326 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335329 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335332 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335334 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335337 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335339 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.346623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335342 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335344 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335347 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335350 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335352 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335355 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335357 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335361 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335363 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335365 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335368 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335371 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335374 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335377 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335380 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335382 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335385 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335387 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335390 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.347165 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335392 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335395 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335397 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335399 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335402 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335404 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335407 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335409 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335412 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335414 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335416 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335419 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335421 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335424 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335426 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335429 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335432 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335434 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335437 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.347609 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335439 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335445 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335447 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335450 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335452 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335455 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335457 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335460 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.335462 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.342186 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.342206 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.342223 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.342228 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342286 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342291 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342296 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342299 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342302 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342305 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342308 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.348102 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342311 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342313 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342316 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342319 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342321 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342323 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342326 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342329 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342331 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342334 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342337 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342340 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342342 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342345 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342348 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342350 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342353 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342356 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342358 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342361 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.348599 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342363 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342366 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342369 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342372 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342375 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342378 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342380 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342383 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342385 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342388 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342390 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342393 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342395 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342398 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342400 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342402 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342405 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342407 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342410 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342412 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.349189 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342415 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342417 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342420 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342423 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342425 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342428 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342430 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342433 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342437 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342440 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342444 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342447 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342450 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342454 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342457 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342461 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342465 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342468 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342471 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.349656 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342474 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342476 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342479 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342482 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342484 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342486 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342489 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342492 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342494 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342496 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342499 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342501 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342504 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342506 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342509 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342512 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342515 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342517 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342520 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.350202 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342522 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342525 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342527 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342530 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342532 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342535 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342538 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342541 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342543 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342546 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342549 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342551 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342554 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342556 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342559 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342561 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342564 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342566 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342568 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.350637 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342571 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.342573 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.342601 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344594 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344612 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344618 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344622 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344626 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344631 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344635 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344641 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344645 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344649 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344653 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344657 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344661 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.351187 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344665 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344673 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344681 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344685 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344703 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344725 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344731 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344735 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344738 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344742 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344747 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344750 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344754 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344759 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344763 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344767 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344772 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344777 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344781 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.351561 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344785 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344789 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344794 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344798 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344802 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344806 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344811 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344815 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344820 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344822 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344825 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344828 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344831 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344833 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344836 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344838 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344841 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344844 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344848 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.352049 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344850 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344853 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344855 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344859 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344862 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344864 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344867 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344869 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344872 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344876 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344881 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344884 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344887 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344889 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344892 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344895 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344898 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344900 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344903 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344906 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.352490 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344909 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344911 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344914 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344916 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344919 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344921 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344924 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344927 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344929 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344932 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344934 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344937 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344940 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344943 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344945 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344948 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344950 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344952 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344955 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.353007 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344957 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344961 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344963 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344966 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344968 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344971 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344974 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344977 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344979 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344982 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344984 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344987 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344989 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344992 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344994 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.344997 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.353435 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.345031 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345070 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345074 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345077 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345080 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345083 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345086 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345089 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345091 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345094 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345097 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345100 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345102 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345105 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345107 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345110 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345113 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345115 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345118 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345120 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:58:07.353851 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345123 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345126 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345128 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345131 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345133 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345136 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345138 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345141 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345144 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345146 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345149 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345152 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345154 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345157 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345159 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345161 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345164 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345167 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345169 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:58:07.354325 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345173 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345178 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345181 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345184 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345187 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345190 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345192 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345195 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345197 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345200 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345203 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345206 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345208 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345211 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345213 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345216 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345221 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345224 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345226 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:58:07.354824 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345229 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345231 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345236 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345239 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345241 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345243 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345246 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345248 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345251 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345253 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345256 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345258 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345261 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345263 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345266 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345268 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345271 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345274 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:58:07.355356 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345276 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345279 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345281 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345284 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345286 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345289 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345291 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345294 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345296 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345299 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345301 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345304 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345306 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345308 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345311 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345314 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345319 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345321 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345324 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345326 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:58:07.355812 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345329 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345333 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345336 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345339 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345342 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345344 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345348 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345351 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:07.345353 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.345420 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.347556 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.348827 2579 server.go:1014] "Starting client certificate rotation" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.348940 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.349738 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.351136 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:07.356276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.352341 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:58:07.359745 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.359703 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:58:07.364841 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.364825 2579 log.go:25] "Validated CRI v1 image API" Oct 30 23:58:07.365605 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.365579 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:58:07.368240 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.368221 2579 fs.go:135] Filesystem UUIDs: map[48e9417f-a2ad-4e02-bfbd-0fa5f0fa70ed:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7696fec-362f-417b-8746-0141c5787a33:/dev/nvme0n1p4] Oct 30 23:58:07.368306 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.368240 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:58:07.378140 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.378002 2579 manager.go:217] Machine: {Timestamp:2026-10-30 23:58:07.376600965 +0000 UTC m=+0.329396761 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090281 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c51afc528781a6d66259fe50a019 SystemUUID:ec22c51a-fc52-8781-a6d6-6259fe50a019 BootID:25cee24e-ac78-44ae-b877-afcf3927b0f4 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:35:15:cb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:35:15:cb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:12:ee:dd:01:44:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:58:07.378564 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.378553 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:58:07.378658 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.378646 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:58:07.381976 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.381944 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:58:07.382116 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.381975 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-236.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:58:07.382163 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.382119 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:58:07.382163 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.382126 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:58:07.382163 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.382142 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:07.382163 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.382158 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:58:07.382918 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.382908 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:58:07.386752 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.386739 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:58:07.387922 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.387897 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:58:07.388011 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.387953 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:58:07.388011 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.387969 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:58:07.388011 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.387987 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:58:07.388011 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.388001 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:58:07.388173 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.388013 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:58:07.388720 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.388670 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:58:07.388807 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.388741 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:07.391426 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.391411 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:58:07.396043 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.396029 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:58:07.396927 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.396915 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:58:07.396966 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.396946 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:58:07.397110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397101 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:58:07.397142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397116 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:58:07.397142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397123 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:58:07.397142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397128 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:58:07.397142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397134 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:58:07.397142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397139 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:58:07.397841 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397833 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:58:07.397872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397842 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:58:07.397872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397849 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:58:07.397872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397855 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:58:07.397872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397864 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:58:07.397872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.397872 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:58:07.398803 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.398794 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:58:07.398842 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.398804 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:58:07.402728 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.402692 2579 server.go:1263] "Started kubelet" Oct 30 23:58:07.402871 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.402826 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:58:07.402871 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.402829 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:58:07.402952 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.402896 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:58:07.403758 ip-10-0-136-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:58:07.404382 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.403930 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:58:07.404382 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.404205 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:58:07.405474 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.405453 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-236.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:58:07.409503 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.409482 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:58:07.409589 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.409535 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:58:07.410055 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410036 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:58:07.410163 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.410137 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.410700 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410674 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:58:07.410700 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410675 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:58:07.410700 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410702 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:58:07.410949 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410843 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:58:07.410949 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.410853 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:58:07.411310 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411297 2579 factory.go:153] Registering CRI-O factory Oct 30 23:58:07.411379 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411313 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 23:58:07.411419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411379 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:58:07.411419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411386 2579 factory.go:55] Registering systemd factory Oct 30 23:58:07.411419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411392 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:58:07.411419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411414 2579 factory.go:103] Registering Raw factory Oct 30 23:58:07.411527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.411424 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 23:58:07.412677 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.412662 2579 manager.go:319] Starting recovery of all containers Oct 30 23:58:07.412885 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.412857 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-236.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:58:07.413498 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.413447 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:58:07.413802 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.412856 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-236.us-east-2.compute.internal.18e3740fb3280592 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-236.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-236.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:58:07.402665362 +0000 UTC m=+0.355461164,LastTimestamp:2026-10-30 23:58:07.402665362 +0000 UTC m=+0.355461164,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-236.us-east-2.compute.internal,}" Oct 30 23:58:07.414121 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.414074 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-236.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:58:07.420987 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.420956 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fng8m" Oct 30 23:58:07.424292 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.424238 2579 manager.go:324] Recovery completed Oct 30 23:58:07.428244 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.428228 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.430128 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430108 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.430196 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430146 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.430196 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430158 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.430663 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430650 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:58:07.430663 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430660 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.430851 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430660 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:58:07.430851 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430695 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:58:07.430851 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.430797 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fng8m" Oct 30 23:58:07.431411 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.431394 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.431472 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.431424 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.431472 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.431434 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.432141 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.432071 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-236.us-east-2.compute.internal.18e3740fb4cb1aaf default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-236.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-236.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-236.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:58:07.430130351 +0000 UTC m=+0.382926150,LastTimestamp:2026-10-30 23:58:07.430130351 +0000 UTC m=+0.382926150,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-236.us-east-2.compute.internal,}" Oct 30 23:58:07.433081 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.433070 2579 policy_none.go:50] "Start" Oct 30 23:58:07.433130 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.433085 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:58:07.433130 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.433095 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:58:07.436846 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.436833 2579 policy_none.go:44] "Start" Oct 30 23:58:07.474528 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.474349 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:58:07.478997 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.478983 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 23:58:07.479077 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.479022 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:58:07.479077 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479032 2579 server.go:87] "Starting device plugin registration server" Oct 30 23:58:07.479273 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479262 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:58:07.479318 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479274 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:58:07.479379 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479364 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:58:07.479480 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479466 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:58:07.479480 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.479481 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:58:07.480105 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.480086 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:58:07.480177 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.480140 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.549092 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.549060 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:58:07.551252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.550452 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:58:07.551252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.550880 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:58:07.551252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.550918 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:58:07.551252 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.550965 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:58:07.554770 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.554754 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:07.579912 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.579826 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.580784 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.580768 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.580871 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.580801 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.580871 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.580814 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.580871 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.580839 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.587467 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.587451 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.587522 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.587480 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-236.us-east-2.compute.internal\": node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.606460 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.606441 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.651097 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.651067 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal"] Oct 30 23:58:07.651193 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.651148 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.652125 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.652111 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.652198 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.652157 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.652198 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.652169 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.654411 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.654397 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.654638 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.654622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.654697 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.654659 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.655246 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655230 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.655318 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655262 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.655318 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655272 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.655318 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655235 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.655432 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655329 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.655432 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.655339 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.657208 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.657191 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.657264 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.657225 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:58:07.658001 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.657979 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:58:07.658103 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.658015 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:58:07.658103 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.658029 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:58:07.681975 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.681944 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.684676 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.684660 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.706528 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.706506 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.712870 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.712840 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.712927 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.712882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.712927 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.712903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.807509 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.807482 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.814062 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814031 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.814062 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.814203 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814089 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.814203 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.814203 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.814203 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.814164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.908548 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:07.908463 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:07.985099 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.985053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:07.988236 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:07.988205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:08.009470 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:08.009446 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:08.110011 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:08.109967 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:08.210455 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:08.210388 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:08.310916 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:08.310890 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:08.348148 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.348131 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:58:08.348557 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.348259 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:58:08.411094 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:08.411064 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:58:08.422432 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.422407 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:08.422532 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.422476 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:08.434902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.434863 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:53:08 +0000 UTC" deadline="2028-08-13 04:06:24.877079338 +0000 UTC" Oct 30 23:58:08.434902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.434899 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15652h8m16.442182738s" Oct 30 23:58:08.511239 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.511183 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:08.521788 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.521769 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:08.523620 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.523607 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:58:08.533901 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.533880 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:58:08.667954 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:08.667915 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d17d96d9d9d049ba1dc638bf8eda6fd.slice/crio-d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052 WatchSource:0}: Error finding container d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052: Status 404 returned error can't find the container with id d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052 Oct 30 23:58:08.668315 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:08.668296 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba88490f41dac4679774c025c13b2ce.slice/crio-41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f WatchSource:0}: Error finding container 41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f: Status 404 returned error can't find the container with id 41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f Oct 30 23:58:08.673392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.673374 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:58:08.785076 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:08.784992 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:09.390173 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.390137 2579 apiserver.go:52] "Watching apiserver" Oct 30 23:58:09.398422 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.398394 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:58:09.398823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.398776 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-2wlm9","openshift-cluster-node-tuning-operator/tuned-t77d6","openshift-multus/network-metrics-daemon-8brkm","openshift-network-operator/iptables-alerter-xf9b7","kube-system/global-pull-secret-syncer-zqb2v","openshift-dns/node-resolver-mmztz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal","openshift-multus/multus-lrwwf","kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal","openshift-image-registry/node-ca-7k929","openshift-ovn-kubernetes/ovnkube-node-t2vhd","kube-system/konnectivity-agent-hbqzw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4","openshift-multus/multus-additional-cni-plugins-cjdd8"] Oct 30 23:58:09.401834 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.401813 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.404295 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.404271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.406466 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.406442 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:58:09.406571 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.406467 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.406642 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.406592 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:58:09.406700 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.406665 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.406700 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.406679 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.406832 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.406759 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:09.407759 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.407730 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:58:09.407920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.407902 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.408160 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.408141 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:58:09.408248 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.408233 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.409928 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.409907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.410543 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.410522 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:58:09.412041 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.412018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.412153 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.412087 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:09.414993 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.414973 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.415689 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.415095 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:58:09.415689 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.415180 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.424432 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.424407 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427354 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlx47\" (UniqueName: \"kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427428 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts78k\" (UniqueName: \"kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427493 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbtk9\" (UniqueName: \"kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427556 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdxzw\" (UniqueName: \"kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.428556 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.429248 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427823 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.429248 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.427857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.429248 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.428954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.429638 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.429613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.430185 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.429946 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:58:09.431291 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.431266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:09.431385 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.431351 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:09.433783 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.433765 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.436300 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.436278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.452477 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.451050 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-67gqj" Oct 30 23:58:09.464303 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.463992 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-67gqj" Oct 30 23:58:09.512177 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.512146 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:58:09.518368 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518322 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.518368 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518335 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:58:09.518858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518639 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.518858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518639 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.518858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518664 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.518858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518799 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:58:09.518858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518829 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.519119 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518863 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:58:09.519119 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.518868 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:58:09.520811 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.519701 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.520811 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.519905 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:58:09.520811 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.520644 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:58:09.520811 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.520743 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:58:09.521162 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521140 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:58:09.521263 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521214 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.521263 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521232 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:58:09.521392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521268 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:58:09.521392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521286 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:58:09.521392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521305 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:58:09.521392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521341 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:58:09.521392 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521350 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:58:09.521647 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521398 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:58:09.521647 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521421 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:58:09.521647 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.521468 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:58:09.522024 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.522003 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:58:09.522524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.522503 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:58:09.528425 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.528533 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.528533 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.528649 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmw27\" (UniqueName: \"kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.528649 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.528759 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.528759 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pggm\" (UniqueName: \"kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.528841 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxzw\" (UniqueName: \"kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.528986 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.528953 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjlgk\" (UniqueName: \"kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529053 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529107 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.529317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529285 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529366 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts78k\" (UniqueName: \"kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2wt\" (UniqueName: \"kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.529633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529623 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529723 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmj5\" (UniqueName: \"kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529800 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529916 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.529963 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.529978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530003 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530059 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530084 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.530271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530268 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530321 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtk9\" (UniqueName: \"kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530381 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530410 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530441 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530518 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530576 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530602 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530628 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530680 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530707 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx47\" (UniqueName: \"kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.530823 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530866 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530920 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.530947 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531126 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.531165 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531186 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531212 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531238 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.531394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.531697 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:09.531697 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.531697 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.531697 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.531886 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531864 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.531947 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531886 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.531947 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.531925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.532052 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.531997 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:09.532109 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.532062 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:10.032041619 +0000 UTC m=+2.984837423 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:09.532294 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532271 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.532368 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532353 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.532468 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532452 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.532526 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532507 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.532692 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532677 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:09.532774 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.532756 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:09.532862 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.532827 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:10.032811339 +0000 UTC m=+2.985607125 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:09.532948 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.532928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.533075 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.533035 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.533140 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.533101 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.541773 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.541622 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:58:09.547130 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.547103 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxzw\" (UniqueName: \"kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.549099 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.549076 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts78k\" (UniqueName: \"kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.550152 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.550125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtk9\" (UniqueName: \"kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:09.551915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.551889 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx47\" (UniqueName: \"kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.562308 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.562260 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"bba88490f41dac4679774c025c13b2ce","Type":"ContainerStarted","Data":"41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f"} Oct 30 23:58:09.564131 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.564097 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerStarted","Data":"d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052"} Oct 30 23:58:09.608301 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.608270 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:58:09.631935 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.631901 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.631944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.631977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632000 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw27\" (UniqueName: \"kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632052 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pggm\" (UniqueName: \"kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlgk\" (UniqueName: \"kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632128 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632144 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.632164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632168 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632197 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632289 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2wt\" (UniqueName: \"kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632307 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632338 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632649 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632662 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632732 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmj5\" (UniqueName: \"kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632817 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632894 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632925 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632956 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.632985 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.632986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633017 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633116 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633171 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633186 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633202 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633310 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633333 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.633509 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633373 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.633763 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.633763 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633902 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633926 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633953 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.633975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634153 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634249 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634302 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634395 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634472 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634513 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634572 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.635178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634588 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634603 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634672 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634684 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634769 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634810 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634839 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634844 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634873 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.634961 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635023 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635040 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635055 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635156 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635248 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635296 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635336 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635582 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635666 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635912 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635977 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.636833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.635985 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636015 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636123 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636140 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636265 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636317 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.636704 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.637366 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.637304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.638301 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.638277 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.638399 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.638303 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.647131 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.646709 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:09.647131 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.646762 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:09.647131 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.646777 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:09.647131 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:09.646865 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:10.146834963 +0000 UTC m=+3.099630769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:09.648065 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.648018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmj5\" (UniqueName: \"kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.648445 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.648410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlgk\" (UniqueName: \"kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.648648 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.648621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw27\" (UniqueName: \"kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.648847 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.648690 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pggm\" (UniqueName: \"kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.650125 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.650081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2wt\" (UniqueName: \"kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.720830 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.720792 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7k929" Oct 30 23:58:09.733962 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.733937 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bd67fa_8a46_4098_a7b2_0dab23bf0da1.slice/crio-17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7 WatchSource:0}: Error finding container 17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7: Status 404 returned error can't find the container with id 17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7 Oct 30 23:58:09.738333 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.738312 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:58:09.743128 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.743107 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:58:09.748257 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.748230 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc869979e_11d8_4619_8c86_04d2b4acedd6.slice/crio-9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a WatchSource:0}: Error finding container 9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a: Status 404 returned error can't find the container with id 9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a Oct 30 23:58:09.749163 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.749140 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mmztz" Oct 30 23:58:09.753661 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.753638 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4382ac5_fea6_4cf2_b1a6_c06514792224.slice/crio-f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f WatchSource:0}: Error finding container f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f: Status 404 returned error can't find the container with id f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f Oct 30 23:58:09.758327 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.758308 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lrwwf" Oct 30 23:58:09.760124 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.760099 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc77441_8a6d_4a2a_95ca_553f8870b21c.slice/crio-d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c WatchSource:0}: Error finding container d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c: Status 404 returned error can't find the container with id d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c Oct 30 23:58:09.768481 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.768459 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2026287d_f8ba_4ba5_9681_d6d74a7d1058.slice/crio-a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301 WatchSource:0}: Error finding container a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301: Status 404 returned error can't find the container with id a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301 Oct 30 23:58:09.769649 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.769628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:58:09.796448 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.796424 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:58:09.799632 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.799609 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:09.808623 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.808595 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36bdf75e_c844_4a33_b5dd_ac33878a0186.slice/crio-c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83 WatchSource:0}: Error finding container c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83: Status 404 returned error can't find the container with id c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83 Oct 30 23:58:09.810389 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.810365 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118f5c91_d3a9_434b_b931_89be645e00e2.slice/crio-575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2 WatchSource:0}: Error finding container 575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2: Status 404 returned error can't find the container with id 575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2 Oct 30 23:58:09.817995 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:09.817970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:09.829661 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:58:09.829629 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99dfb869_c9bf_41d2_832a_d69ae57a802b.slice/crio-b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e WatchSource:0}: Error finding container b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e: Status 404 returned error can't find the container with id b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.039604 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.039704 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.039762 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.039813 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.039838 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:11.039819361 +0000 UTC m=+3.992615147 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:10.039920 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.039859 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:11.039848282 +0000 UTC m=+3.992644069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:10.241334 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.241292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:10.241537 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.241516 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:10.241610 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.241542 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:10.241610 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.241553 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:10.241727 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.241611 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:11.24159605 +0000 UTC m=+4.194392062 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:10.468953 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.468851 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:10 +0000 UTC" deadline="2028-04-05 19:25:28.58862364 +0000 UTC" Oct 30 23:58:10.468953 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.468913 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12547h27m18.1197141s" Oct 30 23:58:10.551506 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.551477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:10.551657 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:10.551630 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:10.570477 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.570375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t77d6" event={"ID":"ec194b59-61ad-4c7f-b93c-1417107e0269","Type":"ContainerStarted","Data":"535857844f7e7581abae842e7cc07e30b01951f74bb8039fb9242ad56e612243"} Oct 30 23:58:10.573929 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.573855 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lrwwf" event={"ID":"2026287d-f8ba-4ba5-9681-d6d74a7d1058","Type":"ContainerStarted","Data":"a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301"} Oct 30 23:58:10.579479 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.579403 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mmztz" event={"ID":"ccc77441-8a6d-4a2a-95ca-553f8870b21c","Type":"ContainerStarted","Data":"d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c"} Oct 30 23:58:10.582050 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.581947 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7k929" event={"ID":"85bd67fa-8a46-4098-a7b2-0dab23bf0da1","Type":"ContainerStarted","Data":"17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7"} Oct 30 23:58:10.584444 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.584320 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e"} Oct 30 23:58:10.593130 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.593092 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf9b7" event={"ID":"b4382ac5-fea6-4cf2-b1a6-c06514792224","Type":"ContainerStarted","Data":"f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f"} Oct 30 23:58:10.596071 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.596042 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" event={"ID":"c869979e-11d8-4619-8c86-04d2b4acedd6","Type":"ContainerStarted","Data":"9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a"} Oct 30 23:58:10.599238 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.599202 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"bba88490f41dac4679774c025c13b2ce","Type":"ContainerStarted","Data":"fe20d33a20f01d6bfbee4acfb501f25d7dd765c74bcbda996527c4401a52605a"} Oct 30 23:58:10.608800 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.608717 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hbqzw" event={"ID":"118f5c91-d3a9-434b-b931-89be645e00e2","Type":"ContainerStarted","Data":"575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2"} Oct 30 23:58:10.618402 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.617518 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" podStartSLOduration=1.617500734 podStartE2EDuration="1.617500734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:10.616939522 +0000 UTC m=+3.569735333" watchObservedRunningTime="2026-10-30 23:58:10.617500734 +0000 UTC m=+3.570296544" Oct 30 23:58:10.621488 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:10.621463 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerStarted","Data":"c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83"} Oct 30 23:58:11.048240 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.048084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:11.048240 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.048146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:11.048484 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.048276 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:11.048484 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.048339 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:13.048320862 +0000 UTC m=+6.001116661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:11.048834 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.048777 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:11.048952 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.048854 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:13.048838248 +0000 UTC m=+6.001634036 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:11.250680 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.249993 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:11.250680 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.250235 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:11.250680 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.250263 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:11.250680 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.250277 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:11.250680 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.250336 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:13.250318047 +0000 UTC m=+6.203113833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:11.469833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.469753 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:10 +0000 UTC" deadline="2028-04-01 02:29:35.342024725 +0000 UTC" Oct 30 23:58:11.469833 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.469796 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12434h31m23.872235104s" Oct 30 23:58:11.555296 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.554769 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:11.555296 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.554922 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:11.555296 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.555002 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:11.555296 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:11.555120 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:11.636421 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.635675 2579 generic.go:356] "Generic (PLEG): container finished" podID="3d17d96d9d9d049ba1dc638bf8eda6fd" containerID="20f9390a59ceb1c61fe2e7724434ea46acfda811348998c823e582a8b7d4c5f2" exitCode=0 Oct 30 23:58:11.636421 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:11.636261 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerDied","Data":"20f9390a59ceb1c61fe2e7724434ea46acfda811348998c823e582a8b7d4c5f2"} Oct 30 23:58:12.552213 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:12.551652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:12.552213 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:12.551823 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:12.641852 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:12.641379 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerStarted","Data":"33cb4d68d59200951e35b137c17895fa81224144ba2cd6a86c2dca03224510f4"} Oct 30 23:58:12.661868 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:12.661320 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" podStartSLOduration=3.661301392 podStartE2EDuration="3.661301392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:12.66025863 +0000 UTC m=+5.613054441" watchObservedRunningTime="2026-10-30 23:58:12.661301392 +0000 UTC m=+5.614097202" Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.065629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.065700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.065887 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.065953 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.065958 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.065937888 +0000 UTC m=+10.018733681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:13.066502 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.066049 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.066028042 +0000 UTC m=+10.018823826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:13.267487 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.267406 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:13.267673 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.267577 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:13.267673 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.267593 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:13.267673 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.267602 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:13.267673 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.267646 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:17.267633705 +0000 UTC m=+10.220429490 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:13.551992 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.551910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:13.551992 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.551962 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:13.552288 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.552075 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:13.552882 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:13.552519 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:13.646524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.646484 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="c6313539a576676bff78cae57898da0693bed7ad6c1cde25b76be924574e2137" exitCode=0 Oct 30 23:58:13.647587 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:13.647527 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"c6313539a576676bff78cae57898da0693bed7ad6c1cde25b76be924574e2137"} Oct 30 23:58:14.551688 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:14.551092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:14.551688 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:14.551251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:14.661164 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:14.661117 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-hbqzw" event={"ID":"118f5c91-d3a9-434b-b931-89be645e00e2","Type":"ContainerStarted","Data":"a9e25de8d21b6d64db8bcba2e23b1d87f5622e9ffe038ccf8fa1a75c8f67688f"} Oct 30 23:58:14.683334 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:14.683267 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-hbqzw" podStartSLOduration=2.390921853 podStartE2EDuration="6.683244942s" totalImagesPullingTime="4.292323089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:14.683219816 +0000 UTC m=+7.636015625" watchObservedRunningTime="2026-10-30 23:58:14.683244942 +0000 UTC m=+7.636040751" Oct 30 23:58:15.554578 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:15.554537 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:15.554776 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:15.554680 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:15.555188 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:15.555149 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:15.555299 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:15.555247 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:15.666749 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:15.666662 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7k929" event={"ID":"85bd67fa-8a46-4098-a7b2-0dab23bf0da1","Type":"ContainerStarted","Data":"dc64df3cd9efa97e2d5350209490004538ea0d093ae967879b3c5c6c10145864"} Oct 30 23:58:15.670644 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:15.670610 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" event={"ID":"c869979e-11d8-4619-8c86-04d2b4acedd6","Type":"ContainerStarted","Data":"ca7d7fa3f4dba2be2041a36d21228b3e061255a73fa1ed36cce638a68b716b08"} Oct 30 23:58:15.685445 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:15.685392 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7k929" podStartSLOduration=2.387512953 podStartE2EDuration="7.685378158s" totalImagesPullingTime="5.297865205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:15.685138814 +0000 UTC m=+8.637934623" watchObservedRunningTime="2026-10-30 23:58:15.685378158 +0000 UTC m=+8.638173977" Oct 30 23:58:16.551357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:16.551319 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:16.551532 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:16.551465 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.101501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.101579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.101787 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.101863 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.101842964 +0000 UTC m=+18.054638773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.101991 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:17.102271 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.102080 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.102059124 +0000 UTC m=+18.054854932 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:17.185838 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.185656 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:58:17.303915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.303816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:17.304091 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.304045 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:17.304091 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.304077 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:17.304091 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.304091 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:17.304250 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.304150 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:25.304136763 +0000 UTC m=+18.256932563 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:17.420364 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.419790 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:17.420515 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.420400 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:17.486216 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.485430 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:58:17.185674954Z","UUID":"7f3c4db3-ed5e-4dd7-a9a7-1d299ae9077e","Handler":null,"Name":"","Endpoint":""} Oct 30 23:58:17.493033 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.492981 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:58:17.493493 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.493204 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:58:17.554352 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.554267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:17.554509 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.554373 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:17.554715 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.554696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:17.554844 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:17.554779 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:17.677984 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.677437 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lrwwf" event={"ID":"2026287d-f8ba-4ba5-9681-d6d74a7d1058","Type":"ContainerStarted","Data":"7d07448814ba75b878586e6c26bbfe77e9c6c52ee1674710adebbe16703bd8d7"} Oct 30 23:58:17.682110 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.682074 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" event={"ID":"c869979e-11d8-4619-8c86-04d2b4acedd6","Type":"ContainerStarted","Data":"b407ba2abeba5f36e6bcf7dc9df8f86adcd42c031da5a297982f5534cd4b02af"} Oct 30 23:58:17.682275 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.682155 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:17.683324 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.683296 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:58:17.700467 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:17.700400 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lrwwf" podStartSLOduration=2.020314426 podStartE2EDuration="9.700382019s" totalImagesPullingTime="7.680067593s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:17.699674362 +0000 UTC m=+10.652470173" watchObservedRunningTime="2026-10-30 23:58:17.700382019 +0000 UTC m=+10.653177828" Oct 30 23:58:18.552403 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.552376 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:18.552893 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:18.552494 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:18.685412 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.685327 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="8aa1e6d28566d17924c42400e88b827bc3051e1b285d0c27b651e9fa385ef423" exitCode=0 Oct 30 23:58:18.685575 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.685421 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"8aa1e6d28566d17924c42400e88b827bc3051e1b285d0c27b651e9fa385ef423"} Oct 30 23:58:18.687066 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.687044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mmztz" event={"ID":"ccc77441-8a6d-4a2a-95ca-553f8870b21c","Type":"ContainerStarted","Data":"4395a4420ba17f6418243733bb22be6b9c9c722c07d13c0e39c2a13f75bfa785"} Oct 30 23:58:18.689194 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.689162 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" event={"ID":"c869979e-11d8-4619-8c86-04d2b4acedd6","Type":"ContainerStarted","Data":"7ac5d41111c39f811115ec1db937f634c35199754b94dd7d6343167e6e5f3283"} Oct 30 23:58:18.744459 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.744405 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" podStartSLOduration=2.5311563599999998 podStartE2EDuration="10.744391617s" totalImagesPullingTime="8.213235257s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:18.744106853 +0000 UTC m=+11.696902658" watchObservedRunningTime="2026-10-30 23:58:18.744391617 +0000 UTC m=+11.697187421" Oct 30 23:58:18.761650 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:18.761606 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-mmztz" podStartSLOduration=2.892469079 podStartE2EDuration="10.761591835s" totalImagesPullingTime="7.869122756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:18.761303337 +0000 UTC m=+11.714099141" watchObservedRunningTime="2026-10-30 23:58:18.761591835 +0000 UTC m=+11.714387637" Oct 30 23:58:19.551741 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.551542 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:19.551822 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.551568 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:19.551870 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:19.551849 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:19.551901 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:19.551882 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:19.692589 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.692497 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xf9b7" event={"ID":"b4382ac5-fea6-4cf2-b1a6-c06514792224","Type":"ContainerStarted","Data":"1001799c7816dc3a2287e2fa3aec18f95144c28a7f37d9b2a5ba9602c43ae182"} Oct 30 23:58:19.694616 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.694588 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="afee42a100231180a0b1dfa714ad882b16493c21a0cac9627233491ea96cbdf4" exitCode=0 Oct 30 23:58:19.694753 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.694702 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"afee42a100231180a0b1dfa714ad882b16493c21a0cac9627233491ea96cbdf4"} Oct 30 23:58:19.708542 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:19.708502 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xf9b7" podStartSLOduration=3.900981365 podStartE2EDuration="11.708488976s" totalImagesPullingTime="7.807507611s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:19.708064192 +0000 UTC m=+12.660859997" watchObservedRunningTime="2026-10-30 23:58:19.708488976 +0000 UTC m=+12.661284781" Oct 30 23:58:20.552098 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:20.552062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:20.552328 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:20.552200 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:21.552056 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:21.552015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:21.552625 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:21.552145 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:21.552625 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:21.552202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:21.552625 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:21.552287 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:21.702552 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:21.702516 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="d30ed5c5327c9eac96479b0b48f9a7a57710b69d5a6ad4b0393389e1f87e5597" exitCode=0 Oct 30 23:58:21.702752 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:21.702564 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"d30ed5c5327c9eac96479b0b48f9a7a57710b69d5a6ad4b0393389e1f87e5597"} Oct 30 23:58:22.552394 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:22.552320 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:22.552999 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:22.552443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:22.706104 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:22.706059 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t77d6" event={"ID":"ec194b59-61ad-4c7f-b93c-1417107e0269","Type":"ContainerStarted","Data":"d97b33de1854a1349a9cd47dd58aaf1fe6041eae4fa8e8e56443372bc695b0ee"} Oct 30 23:58:22.726190 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:22.726135 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t77d6" podStartSLOduration=2.369425724 podStartE2EDuration="14.726121746s" totalImagesPullingTime="12.356696022s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:22.725853249 +0000 UTC m=+15.678649055" watchObservedRunningTime="2026-10-30 23:58:22.726121746 +0000 UTC m=+15.678917556" Oct 30 23:58:23.552316 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:23.552112 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:23.552478 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:23.552162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:23.552478 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:23.552416 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:23.552478 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:23.552470 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:23.845616 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:23.845487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:24.030878 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:24.030846 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:24.551481 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:24.551403 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:24.551640 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:24.551534 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:24.712056 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:24.712027 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="54f61acdd5c7af30098f64284b29f25a5ae670fe846814ac81d169f4eabee118" exitCode=0 Oct 30 23:58:24.712473 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:24.712082 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"54f61acdd5c7af30098f64284b29f25a5ae670fe846814ac81d169f4eabee118"} Oct 30 23:58:25.169257 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.169183 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:25.169257 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.169240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:25.169426 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.169328 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:25.169426 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.169328 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:25.169426 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.169388 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:41.169372878 +0000 UTC m=+34.122168676 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:25.169426 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.169405 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:58:41.169396601 +0000 UTC m=+34.122192399 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:25.369900 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.369874 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:25.370065 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.369985 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:25.370065 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.370003 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:25.370065 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.370014 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:25.370065 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.370057 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:58:41.370045459 +0000 UTC m=+34.322841242 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:25.551342 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.551266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:25.551469 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.551274 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:25.551469 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.551368 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:25.551469 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:25.551433 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:25.717267 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.717230 2579 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="62dfe26f7587906c427faeefd1fa19909ce0115de036cb84863d1b8d9ca6292d" exitCode=0 Oct 30 23:58:25.717605 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:25.717281 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerDied","Data":"62dfe26f7587906c427faeefd1fa19909ce0115de036cb84863d1b8d9ca6292d"} Oct 30 23:58:26.552100 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:26.552071 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:26.552248 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:26.552174 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:26.722724 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:26.722673 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" event={"ID":"36bdf75e-c844-4a33-b5dd-ac33878a0186","Type":"ContainerStarted","Data":"c11bbbfebede09ee4130932ef4512705634ab32eab950bfc3c45ee9d995528ce"} Oct 30 23:58:26.755836 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:26.755793 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cjdd8" podStartSLOduration=7.505695286 podStartE2EDuration="18.755774317s" totalImagesPullingTime="11.250079031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:26.755202292 +0000 UTC m=+19.707998098" watchObservedRunningTime="2026-10-30 23:58:26.755774317 +0000 UTC m=+19.708570124" Oct 30 23:58:27.552108 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:27.552078 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:27.552267 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:27.552161 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:27.552267 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:27.552244 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:27.552400 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:27.552345 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:27.747064 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:27.747035 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:27.905222 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:27.905151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:28.551875 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:28.551849 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:28.552077 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:28.551964 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:29.551554 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:29.551521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:29.551554 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:29.551552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:29.552025 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:29.551627 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:29.552025 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:29.551702 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:29.980508 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:29.980425 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:30.115654 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:30.115626 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:30.314144 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:30.314090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:30.552033 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:30.552003 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:30.552455 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:30.552119 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:30.757585 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:30.757539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:30.888662 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:30.888628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:31.047510 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:31.047430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:31.187764 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:31.187733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:31.321186 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:31.321119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:31.551662 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:31.551630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:31.551829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:31.551632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:31.551829 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:31.551748 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:31.551829 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:31.551797 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:32.551943 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:32.551903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:32.552310 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:32.552024 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:32.567866 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:32.567837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.693631 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:32.693591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:32.860413 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:32.860330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:33.006636 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:33.006605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:33.552109 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:33.552077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:33.552109 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:33.552116 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:33.552531 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:33.552218 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:33.552531 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:33.552307 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:33.658960 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:33.658920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.233341 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.233311 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mmztz_ccc77441-8a6d-4a2a-95ca-553f8870b21c/dns-node-resolver/0.log" Oct 30 23:58:34.260661 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.260631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.477872 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.477812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:34.551974 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.551950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:34.552144 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:34.552053 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:34.832812 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.832743 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7k929_85bd67fa-8a46-4098-a7b2-0dab23bf0da1/node-ca/0.log" Oct 30 23:58:34.860230 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:34.860196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:35.066513 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:35.066480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:35.544675 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:35.544637 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:35.551856 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:35.551832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:35.551991 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:35.551957 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:35.552050 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:35.552035 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:35.552147 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:35.552122 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:35.679220 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:35.679187 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:36.259376 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:36.259344 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:36.476682 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:36.476644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:36.551881 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:36.551819 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:36.552022 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:36.551922 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:36.863237 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:36.863153 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.106964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:37.106929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.241133 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:37.241057 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.455041 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:37.455003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:37.552717 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:37.552681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:37.552904 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:37.552871 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:37.552991 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:37.552919 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:37.553036 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:37.552993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:38.089185 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.089150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.221156 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.221093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.357938 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.357850 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.492210 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.492177 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.551737 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.551688 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:38.551923 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:38.551855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:38.784144 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.784115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:38.905465 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:38.905433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.068576 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.068490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.235899 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.235854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.420583 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.420491 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.549265 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.549227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:39.551312 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.551292 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:39.551436 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.551345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:39.551436 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:39.551424 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:39.551774 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:39.551744 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:39.678405 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:39.678287 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:40.067702 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.067664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:40.200156 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.200118 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:40.331021 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.330952 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:40.453314 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.453283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:40.551786 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.551753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:40.551934 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:40.551875 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:40.863029 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:40.862997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:41.079420 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.079382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:41.192038 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.191859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:41.192129 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.192077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:41.192129 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.191995 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:41.192195 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.192179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs podName:a8f07f34-fa12-448b-8054-910fcac344a5 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:13.192160163 +0000 UTC m=+66.144955949 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs") pod "network-metrics-daemon-8brkm" (UID: "a8f07f34-fa12-448b-8054-910fcac344a5") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:58:41.192195 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.192186 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:41.192271 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.192233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret podName:dfd6c668-59d3-4840-8ccc-e4e9c7066b44 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:13.192218315 +0000 UTC m=+66.145014101 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret") pod "global-pull-secret-syncer-zqb2v" (UID: "dfd6c668-59d3-4840-8ccc-e4e9c7066b44") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:58:41.392864 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.392784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:41.393276 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.392942 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:58:41.393276 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.392959 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:58:41.393276 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.392970 2579 projected.go:196] Error preparing data for projected volume kube-api-access-m4598 for pod openshift-network-diagnostics/network-check-target-2wlm9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:41.393276 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.393023 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598 podName:02b16bc6-d881-4af8-b248-72a6455b35cd nodeName:}" failed. No retries permitted until 2026-10-30 23:59:13.393009819 +0000 UTC m=+66.345805625 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m4598" (UniqueName: "kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598") pod "network-check-target-2wlm9" (UID: "02b16bc6-d881-4af8-b248-72a6455b35cd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:58:41.551278 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.551245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:41.551422 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.551366 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:41.551422 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.551414 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:41.551516 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:41.551500 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:41.655305 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.655232 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:41.754390 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754366 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:58:41.754764 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754744 2579 generic.go:356] "Generic (PLEG): container finished" podID="99dfb869-c9bf-41d2-832a-d69ae57a802b" containerID="f544c44d1d90495b164b0e62fb31a120e88341d8291e05136fb77d616c45f80c" exitCode=1 Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754782 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"e4f2e9a7ec3042298dd90fd515e059ea0b6f094168e630c5cb8fb97bd4e6d11c"} Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754831 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"5d25d08cc0c80a0d065086d5220b8223d17834fde0b0393cd5b10875371d34af"} Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754862 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"95a6edee141db6da7135fea52dc74d6c2c9f80ea4017fcae87a2c3f25f8b05f3"} Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"31049818e6f478737c1436414cb4a913bcfceab52599ad09b49769fd1af3dc1f"} Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754916 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerDied","Data":"f544c44d1d90495b164b0e62fb31a120e88341d8291e05136fb77d616c45f80c"} Oct 30 23:58:41.754964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:41.754942 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"ef92a9f958fabf769fc7fab8c8565ab4d58f939f25d94f3688a9e4f1d5a6f354"} Oct 30 23:58:42.261678 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:42.261647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:42.477419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:42.477389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:42.551910 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:42.551818 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:42.552070 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:42.551948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:43.551873 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.551841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:43.552256 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:43.551966 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:43.552256 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.551995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:43.552256 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:43.552068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:43.723867 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.723764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:43.761141 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.761114 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:58:43.761504 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.761484 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"574212858a50bb0d82face64ee16caaf822a4ff99ee40bc59823acbc86f65b47"} Oct 30 23:58:43.850489 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:43.850459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:44.067971 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:44.067934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:44.546116 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:44.546077 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:44.552052 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:44.552032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:44.552324 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:44.552135 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:45.551690 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.551611 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:45.551846 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.551648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:45.551981 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:45.551955 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:45.552113 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:45.552089 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:45.768901 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.768877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:58:45.769253 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.769233 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"acde366e327e5227340c579c913fec7b19b28b9d9d904d9bd60c8197c43e34d8"} Oct 30 23:58:45.769508 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.769492 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:45.769566 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.769558 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:45.769613 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.769598 2579 scope.go:122] "RemoveContainer" containerID="f544c44d1d90495b164b0e62fb31a120e88341d8291e05136fb77d616c45f80c" Oct 30 23:58:45.785694 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:45.785673 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:46.551386 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.551350 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:46.551554 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:46.551464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:46.774641 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.774614 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:58:46.775166 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.774948 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" event={"ID":"99dfb869-c9bf-41d2-832a-d69ae57a802b","Type":"ContainerStarted","Data":"e5afa4427dcb4c8b47152ff63b7142b5b38cd829e86cd9a7b93691d3a06eacd5"} Oct 30 23:58:46.777996 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.775332 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:46.793131 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.793104 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:58:46.836154 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:46.836045 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" podStartSLOduration=7.83543254 podStartE2EDuration="38.836031482s" totalImagesPullingTime="31.000598942s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:46.835530168 +0000 UTC m=+39.788325972" watchObservedRunningTime="2026-10-30 23:58:46.836031482 +0000 UTC m=+39.788827288" Oct 30 23:58:47.552311 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:47.552134 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:47.552506 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:47.552205 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:47.552506 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:47.552372 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:47.552506 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:47.552452 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:47.635087 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:47.635037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:47.769859 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:47.769822 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:48.551212 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:48.551168 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:48.551639 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:48.551281 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:49.552119 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:49.552088 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:49.552119 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:49.552120 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:49.552507 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:49.552192 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:49.552507 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:49.552280 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:50.551950 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:50.551904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:50.552128 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:50.552022 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:51.551616 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:51.551585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:51.552099 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:51.551694 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:51.552099 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:51.551766 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:51.552099 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:51.551850 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:52.552059 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:52.552031 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:52.552425 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:52.552130 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:53.508371 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:53.508331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:53.551599 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:53.551577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:53.551599 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:53.551593 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:53.551741 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:53.551683 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:53.551826 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:53.551806 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:53.664141 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:53.664109 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:53.863029 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:53.862942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:54.209345 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.209279 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wlm9"] Oct 30 23:58:54.209514 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.209392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:54.209514 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:54.209487 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:54.211928 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.211870 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8brkm"] Oct 30 23:58:54.212246 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.212227 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:54.212364 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:54.212318 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:54.212544 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.212525 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqb2v"] Oct 30 23:58:54.212629 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:54.212616 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:54.212744 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:54.212724 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:55.551912 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:55.551874 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:55.552290 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:55.551998 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:56.423693 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:56.423665 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:56.548276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:56.548245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:56.551533 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:56.551512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:56.551663 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:56.551550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:56.551663 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:56.551626 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:56.551975 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:56.551751 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:57.334255 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.334205 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:57.471476 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.471440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:57.553591 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.553563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:57.554043 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:57.553702 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqb2v" podUID="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" Oct 30 23:58:57.654507 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.654426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:57.819010 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.818980 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:57.951477 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:57.951380 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:58.097274 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:58.097236 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:58.552086 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:58.552015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:58:58.552216 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:58.552136 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8brkm" podUID="a8f07f34-fa12-448b-8054-910fcac344a5" Oct 30 23:58:58.552216 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:58.552182 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:58:58.552282 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:58:58.552243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2wlm9" podUID="02b16bc6-d881-4af8-b248-72a6455b35cd" Oct 30 23:58:59.381129 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.381103 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeReady" Oct 30 23:58:59.381551 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.381203 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:58:59.444183 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.444154 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gg8bt"] Oct 30 23:58:59.447755 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.447732 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-d8zkg"] Oct 30 23:58:59.447904 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.447846 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.452089 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.452067 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.454198 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.454181 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:58:59.454469 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.454452 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:58:59.454557 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.454451 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:58:59.457023 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.456983 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:58:59.458492 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.458472 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg8bt"] Oct 30 23:58:59.473225 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.473205 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8zkg"] Oct 30 23:58:59.517869 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.517843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.517869 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.517868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.518031 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.517902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.518031 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.517918 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.518031 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.517995 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xb7l\" (UniqueName: \"kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.518180 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.518048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.518180 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.518108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.518180 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.518171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65bn7\" (UniqueName: \"kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.518322 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.518201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.547977 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.547955 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8vlhd"] Oct 30 23:58:59.550912 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.550897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.551111 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.551087 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:58:59.556252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.556157 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:58:59.556252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.556229 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:58:59.556252 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.556233 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:58:59.556466 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.556261 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:58:59.556517 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.556506 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:58:59.557205 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.557185 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:58:59.557205 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.557196 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:58:59.557357 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.557247 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:58:59.557791 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.557776 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:58:59.557889 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.557841 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:58:59.561911 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.561892 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vlhd"] Oct 30 23:58:59.618793 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.618920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.618920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.618920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.618920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.618920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.618912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xb7l\" (UniqueName: \"kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.619150 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619150 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619150 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvjn\" (UniqueName: \"kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.619276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619200 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619374 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619272 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bn7\" (UniqueName: \"kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619374 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619304 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.619374 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619311 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.619595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619576 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.619772 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.619752 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.622725 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.622684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.622835 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.622687 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.629749 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.629702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:59.631334 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.631287 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xb7l\" (UniqueName: \"kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.631441 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.631424 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bn7\" (UniqueName: \"kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.720600 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.720570 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.720742 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.720631 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvjn\" (UniqueName: \"kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.722901 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.722878 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.732868 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.732850 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvjn\" (UniqueName: \"kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:58:59.758538 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.758510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:58:59.763672 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.763657 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gg8bt" Oct 30 23:58:59.772224 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.772208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:58:59.863149 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:58:59.863123 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:59:00.049364 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.049335 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gg8bt"] Oct 30 23:59:00.052214 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:00.052188 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692f415f_94bd_4fd2_ba6c_468145504bc0.slice/crio-a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b WatchSource:0}: Error finding container a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b: Status 404 returned error can't find the container with id a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b Oct 30 23:59:00.057529 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.054942 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-d8zkg"] Oct 30 23:59:00.060786 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:00.060764 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b456bf9_fa54_4740_998e_d0788f096c3c.slice/crio-72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2 WatchSource:0}: Error finding container 72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2: Status 404 returned error can't find the container with id 72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2 Oct 30 23:59:00.131678 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.131658 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8vlhd"] Oct 30 23:59:00.134558 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:00.134525 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c12cca1_08f4_49f6_a54d_a02e676f6b54.slice/crio-d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b WatchSource:0}: Error finding container d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b: Status 404 returned error can't find the container with id d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b Oct 30 23:59:00.148577 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.148546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:00.375139 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.375102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:00.507826 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.507794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:00.551505 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.551474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:00.551505 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.551505 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:00.555917 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.555877 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:00.556849 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.556828 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:00.556968 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.556876 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:00.556968 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.556884 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:00.557068 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.556981 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:00.810075 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.810046 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8zkg" event={"ID":"5b456bf9-fa54-4740-998e-d0788f096c3c","Type":"ContainerStarted","Data":"c6d2369d56b3718e54192cf4cec4aed269f27cfbddde5c9672a8e93a0d9451ba"} Oct 30 23:59:00.810279 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.810104 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8zkg" event={"ID":"5b456bf9-fa54-4740-998e-d0788f096c3c","Type":"ContainerStarted","Data":"72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2"} Oct 30 23:59:00.811061 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.811041 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg8bt" event={"ID":"692f415f-94bd-4fd2-ba6c-468145504bc0","Type":"ContainerStarted","Data":"a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b"} Oct 30 23:59:00.812256 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.812236 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vlhd" event={"ID":"5c12cca1-08f4-49f6-a54d-a02e676f6b54","Type":"ContainerStarted","Data":"d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b"} Oct 30 23:59:00.828088 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.828038 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:00.959977 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:00.959948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:01.816614 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:01.816580 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8zkg" event={"ID":"5b456bf9-fa54-4740-998e-d0788f096c3c","Type":"ContainerStarted","Data":"3aba892a013a38e1e78a9c0999288e4f075e860430ab9c578b24c0b03a4dc2a6"} Oct 30 23:59:02.498915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.498882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:02.645518 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.645433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:02.787015 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.786979 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mddnv"] Oct 30 23:59:02.790298 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.790275 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.821946 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.821915 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-d8zkg" event={"ID":"5b456bf9-fa54-4740-998e-d0788f096c3c","Type":"ContainerStarted","Data":"deaf7319a6a52732be62a014c7183a6c71ddb81f2a38ec75980441b7da5ec94f"} Oct 30 23:59:02.840038 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.840008 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcdsb\"" Oct 30 23:59:02.840176 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.840125 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:59:02.841137 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.841112 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:59:02.842524 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.842461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:02.849530 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849644 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849644 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849644 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849645 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849690 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fv6v\" (UniqueName: \"kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849745 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.849829 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.849764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.871118 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.871092 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:59:02.871240 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.871199 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:59:02.871240 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.871211 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:59:02.871350 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.871320 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:59:02.875680 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.875642 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-d8zkg" podStartSLOduration=1.71571959 podStartE2EDuration="3.875626994s" totalImagesPullingTime="2.159907404s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:02.875081597 +0000 UTC m=+55.827877404" watchObservedRunningTime="2026-10-30 23:59:02.875626994 +0000 UTC m=+55.828422799" Oct 30 23:59:02.950458 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950458 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv6v\" (UniqueName: \"kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950670 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950670 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950810 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950748 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950862 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950912 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950858 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.950912 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.950862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951067 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951261 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951325 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951399 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951378 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951537 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951513 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.951615 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.951571 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.952104 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.952083 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.953478 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.953455 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.954038 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.954020 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.964761 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.964737 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv6v\" (UniqueName: \"kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:02.977152 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:02.977122 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.110064 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.110033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:03.122645 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:03.122613 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ac9fb68_1982_42a6_8a34_7b9fa61a7565.slice/crio-74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf WatchSource:0}: Error finding container 74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf: Status 404 returned error can't find the container with id 74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf Oct 30 23:59:03.131655 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.131624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.379680 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.379646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.571205 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.571179 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.759804 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.759759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.809798 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.809766 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:03.813144 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.813124 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.817146 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.817118 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:59:03.817772 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.817750 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818163 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818220 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818232 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818238 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818263 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f8xwm\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818276 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:59:03.818321 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818298 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:59:03.818655 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.818353 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:59:03.827690 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.827664 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8vlhd" event={"ID":"5c12cca1-08f4-49f6-a54d-a02e676f6b54","Type":"ContainerStarted","Data":"10e00ee24f649ab932ad8f3eb43bafa08160cafec8f06eed94ddea90b14b140a"} Oct 30 23:59:03.832287 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.832244 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg8bt" event={"ID":"692f415f-94bd-4fd2-ba6c-468145504bc0","Type":"ContainerStarted","Data":"5e96bcc99e65718dc75180faa5dca404970e59cf27af359dcb9707d43823ecea"} Oct 30 23:59:03.832374 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.832297 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gg8bt" event={"ID":"692f415f-94bd-4fd2-ba6c-468145504bc0","Type":"ContainerStarted","Data":"484bcc19f4d06ad0550e2a9ebb8dca66912f725ac24ca9e5d192d5c796a394a0"} Oct 30 23:59:03.833491 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.833474 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:03.834953 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.834923 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mddnv" event={"ID":"9ac9fb68-1982-42a6-8a34-7b9fa61a7565","Type":"ContainerStarted","Data":"74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf"} Oct 30 23:59:03.839499 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.839450 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:03.859201 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859170 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859201 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859513 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859576 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859633 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859627 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859797 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.859908 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.860388 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.860070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.874698 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.874657 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8vlhd" podStartSLOduration=1.9011728300000001 podStartE2EDuration="4.874646986s" totalImagesPullingTime="2.973474156s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:03.874522876 +0000 UTC m=+56.827318678" watchObservedRunningTime="2026-10-30 23:59:03.874646986 +0000 UTC m=+56.827442791" Oct 30 23:59:03.894830 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.894787 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-gg8bt" podStartSLOduration=1.9899137580000001 podStartE2EDuration="4.894779167s" totalImagesPullingTime="2.904865409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:03.894141134 +0000 UTC m=+56.846936940" watchObservedRunningTime="2026-10-30 23:59:03.894779167 +0000 UTC m=+56.847574999" Oct 30 23:59:03.899525 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.899498 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:03.961513 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961478 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961671 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961671 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961603 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961671 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961634 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961671 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.961908 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.961893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.962283 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.962146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.962800 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:59:03.962768 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle podName:0d65c50f-29cb-4be0-8759-8a19c4f1b7d0 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:04.462749064 +0000 UTC m=+57.415544854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0") : configmap references non-existent config key: ca-bundle.crt Oct 30 23:59:03.963583 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.963556 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.964760 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.964697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.965345 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.965307 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.966043 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.966019 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.966241 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.966220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.966737 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.966698 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.967427 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.967384 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.968289 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.968269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.968376 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.968335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.969087 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.969065 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:03.986594 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:03.986568 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:04.462834 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.462801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:04.465775 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.465755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:04.465931 ip-10-0-136-236 kubenswrapper[2579]: E1030 23:59:04.465919 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle podName:0d65c50f-29cb-4be0-8759-8a19c4f1b7d0 nodeName:}" failed. No retries permitted until 2026-10-30 23:59:05.465903197 +0000 UTC m=+58.418698980 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0") : configmap references non-existent config key: ca-bundle.crt Oct 30 23:59:04.667303 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.667271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:04.798595 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.798562 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-84b5d76994-jq7jv"] Oct 30 23:59:04.802462 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.802446 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.810211 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810187 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:59:04.810362 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810227 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:59:04.810362 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810344 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-94v5k\"" Oct 30 23:59:04.810482 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810364 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:59:04.810482 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810368 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:59:04.810482 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810380 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5mnbc9lu3rrpo\"" Oct 30 23:59:04.810482 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.810441 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:59:04.820080 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.820052 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84b5d76994-jq7jv"] Oct 30 23:59:04.840413 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.840378 2579 generic.go:356] "Generic (PLEG): container finished" podID="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" containerID="5e86837f09aaee2771e702ae294b6171a8cbc15fa300494b81f23b5d1390003b" exitCode=0 Oct 30 23:59:04.840843 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.840460 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mddnv" event={"ID":"9ac9fb68-1982-42a6-8a34-7b9fa61a7565","Type":"ContainerDied","Data":"5e86837f09aaee2771e702ae294b6171a8cbc15fa300494b81f23b5d1390003b"} Oct 30 23:59:04.867842 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.867812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.867975 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.867857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.867975 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.867894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868317 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868295 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868377 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868419 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868407 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868486 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868519 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndwrv\" (UniqueName: \"kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.868583 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.868567 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969187 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969304 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969211 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969304 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969243 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwrv\" (UniqueName: \"kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969454 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969526 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969486 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969630 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969690 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969647 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969766 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969698 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969938 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.969938 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.969840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.970992 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.970966 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.972132 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.972110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.972581 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.972536 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.972581 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.972560 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.973093 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.973046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.973093 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.973090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.973238 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.973190 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:04.992139 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:04.992118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwrv\" (UniqueName: \"kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:05.126586 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.126558 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:05.406249 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.406222 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-84b5d76994-jq7jv"] Oct 30 23:59:05.408451 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:05.408428 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aeaa875_fa12_46cc_b25d_453ce96d4a1e.slice/crio-c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7 WatchSource:0}: Error finding container c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7: Status 404 returned error can't find the container with id c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7 Oct 30 23:59:05.474153 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.474125 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:05.475071 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.475050 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:05.639936 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.639907 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:05.820538 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.820507 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:05.845548 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.845511 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mddnv" event={"ID":"9ac9fb68-1982-42a6-8a34-7b9fa61a7565","Type":"ContainerStarted","Data":"949ff43696936c7c5c5816c646f3181607b795db75e253fb56c68dc552c9958b"} Oct 30 23:59:05.845974 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.845573 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mddnv" event={"ID":"9ac9fb68-1982-42a6-8a34-7b9fa61a7565","Type":"ContainerStarted","Data":"158bad3c09b9e4c251b739497bf86a026c6ef60b34e40fbf7c948570e3d08d98"} Oct 30 23:59:05.846662 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.846644 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7"} Oct 30 23:59:05.869790 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.869748 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mddnv" podStartSLOduration=2.689910301 podStartE2EDuration="3.869736691s" totalImagesPullingTime="1.106125124s" totalInitContainerRuntime="73.701266ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:59:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:05.868977669 +0000 UTC m=+58.821773470" watchObservedRunningTime="2026-10-30 23:59:05.869736691 +0000 UTC m=+58.822532489" Oct 30 23:59:05.948978 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:05.948914 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:05.952132 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:05.952111 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d65c50f_29cb_4be0_8759_8a19c4f1b7d0.slice/crio-4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22 WatchSource:0}: Error finding container 4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22: Status 404 returned error can't find the container with id 4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22 Oct 30 23:59:06.022399 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:06.022366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:06.852351 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:06.852315 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22"} Oct 30 23:59:07.514882 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.514851 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9"] Oct 30 23:59:07.517817 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.517798 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.522224 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.522204 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:59:07.522956 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.522936 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nqp6h\"" Oct 30 23:59:07.527396 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.527372 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9"] Oct 30 23:59:07.551097 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.551071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:07.594119 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.594088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d62jc\" (UniqueName: \"kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.594271 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.594149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.694526 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.694445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62jc\" (UniqueName: \"kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.694526 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.694524 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.696966 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.696945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.706468 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.706446 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62jc\" (UniqueName: \"kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.721070 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.721046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:07.830748 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.830701 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:07.857246 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.857213 2579 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" exitCode=0 Oct 30 23:59:07.857622 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.857333 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190"} Oct 30 23:59:07.859680 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.859654 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"e9d106eeef53c9e70bbde77f607843f715061e05d80a63149f02ae54f9ae092e"} Oct 30 23:59:07.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.859742 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"4a8acf909347c08ca12fbb90fc30bfe784d6cd204cc625ffd33caee92c88f067"} Oct 30 23:59:07.859915 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.859781 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"9450a0f795321943970fffb07e75f9bf87b16e9aedc6dcdac42cf80e8a6d1c98"} Oct 30 23:59:07.860069 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:07.859918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:08.118846 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.118332 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9"] Oct 30 23:59:08.121625 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:08.121594 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe424bc5_4967_481f_bc33_3f7b3db28a9e.slice/crio-df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af WatchSource:0}: Error finding container df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af: Status 404 returned error can't find the container with id df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af Oct 30 23:59:08.867936 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.867903 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"00396ae4524e02eac5e80ac524370f7f6ce066f46514734f8d27c7dabd689a63"} Oct 30 23:59:08.868341 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.867978 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"ec6ed086e5399091fff9bdb2f2a847ef78698717c2668dad32d422351b80c047"} Oct 30 23:59:08.868341 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.868017 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" event={"ID":"6aeaa875-fa12-46cc-b25d-453ce96d4a1e","Type":"ContainerStarted","Data":"0d2c692b35e9c1375ceff307b25cb9a216a51a0073cc64799a58ea81b55a1fd8"} Oct 30 23:59:08.868341 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.868081 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:08.869107 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.869082 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" event={"ID":"fe424bc5-4967-481f-bc33-3f7b3db28a9e","Type":"ContainerStarted","Data":"df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af"} Oct 30 23:59:08.897977 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:08.897930 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" podStartSLOduration=2.149993335 podStartE2EDuration="4.897916879s" totalImagesPullingTime="2.747923544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:08.897548858 +0000 UTC m=+61.850344676" watchObservedRunningTime="2026-10-30 23:59:08.897916879 +0000 UTC m=+61.850712683" Oct 30 23:59:09.082917 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.082879 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:09.086680 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.086656 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.097322 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097294 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:59:09.097453 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097363 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:59:09.097636 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097617 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:59:09.097801 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097779 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097807 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097810 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097829 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097818 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fqabajevadsu4\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097830 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097865 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:59:09.097902 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.097888 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:59:09.098862 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.098844 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:59:09.098862 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.098855 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:59:09.099250 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.099233 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2k6kb\"" Oct 30 23:59:09.102142 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.102126 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:59:09.106276 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.105126 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:59:09.106791 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.106770 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:59:09.110787 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.110382 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:09.160249 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.160149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:09.208362 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208329 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208506 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208506 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208590 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208537 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208627 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208627 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208732 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208832 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208744 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208832 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208767 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208900 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208900 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.208960 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209009 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.208994 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209042 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209042 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209120 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209108 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209149 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209218 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209263 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.209301 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.209264 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.309949 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.309921 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.309949 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.309954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310150 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310205 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310161 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310258 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310258 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310234 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310315 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310315 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310283 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310393 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310315 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310393 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310488 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310389 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.310488 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.310414 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311032 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311515 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311658 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.311765 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.312744 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.312499 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.314762 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.314639 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.315750 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.314967 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.315750 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.315692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.316143 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.316110 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.316459 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.316442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.316649 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.316628 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.316811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.316921 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.317039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.317097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.317414 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.317442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.317527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.317489 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.318170 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.318145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.319002 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.318982 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.325303 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.325282 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.325439 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.325416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.417977 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.417903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:09.714527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.714503 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:09.716887 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:09.716858 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6948ba0_e600_4cfe_8e6d_e8e912da4861.slice/crio-d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15 WatchSource:0}: Error finding container d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15: Status 404 returned error can't find the container with id d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15 Oct 30 23:59:09.873421 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.873382 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" event={"ID":"fe424bc5-4967-481f-bc33-3f7b3db28a9e","Type":"ContainerStarted","Data":"1f61ad94fa5f29f987cdf819e6ec1d22f24917ee9e0061da1d811972df520f7c"} Oct 30 23:59:09.873858 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.873690 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:09.875279 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.875254 2579 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" exitCode=0 Oct 30 23:59:09.875404 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.875342 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} Oct 30 23:59:09.875464 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.875419 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15"} Oct 30 23:59:09.879234 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.879216 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:09.891958 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:09.891878 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" podStartSLOduration=1.408907291 podStartE2EDuration="2.891866185s" totalImagesPullingTime="1.482958894s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:09.891372379 +0000 UTC m=+62.844168184" watchObservedRunningTime="2026-10-30 23:59:09.891866185 +0000 UTC m=+62.844661989" Oct 30 23:59:10.229238 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:10.229195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:11.886841 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886768 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5"} Oct 30 23:59:11.886841 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886834 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb"} Oct 30 23:59:11.887208 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886863 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed"} Oct 30 23:59:11.887208 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886888 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e"} Oct 30 23:59:11.887208 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886912 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2"} Oct 30 23:59:11.887208 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.886934 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerStarted","Data":"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154"} Oct 30 23:59:11.926476 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:11.926425 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.610242714 podStartE2EDuration="8.926405739s" totalImagesPullingTime="4.304999059s" totalInitContainerRuntime="11.163966ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:11.924405182 +0000 UTC m=+64.877200987" watchObservedRunningTime="2026-10-30 23:59:11.926405739 +0000 UTC m=+64.879201545" Oct 30 23:59:12.057035 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:12.057000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:12.185973 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:12.185898 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:13.232570 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.232530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:13.249526 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.249495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:13.249643 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.249550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:13.252256 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.252238 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:13.253282 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.253262 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:59:13.262568 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.262547 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:13.262679 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.262659 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:13.358962 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.358921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:13.371964 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.371940 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:59:13.379184 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.379165 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:13.451998 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.451759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:13.455196 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.455156 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:13.472987 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.472967 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:13.475621 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.475586 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:13.480662 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.480638 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:13.483596 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.483557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:13.682882 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.682850 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqb2v"] Oct 30 23:59:13.683988 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.683776 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:13.686173 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:13.686139 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfd6c668_59d3_4840_8ccc_e4e9c7066b44.slice/crio-a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56 WatchSource:0}: Error finding container a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56: Status 404 returned error can't find the container with id a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56 Oct 30 23:59:13.766239 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.766222 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:13.773840 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.773768 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:13.815797 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.815775 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8brkm"] Oct 30 23:59:13.819278 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:13.819254 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f07f34_fa12_448b_8054_910fcac344a5.slice/crio-143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9 WatchSource:0}: Error finding container 143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9: Status 404 returned error can't find the container with id 143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9 Oct 30 23:59:13.819515 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.819492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:13.843494 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.843470 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:13.898213 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.896740 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqb2v" event={"ID":"dfd6c668-59d3-4840-8ccc-e4e9c7066b44","Type":"ContainerStarted","Data":"a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56"} Oct 30 23:59:13.900197 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900173 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} Oct 30 23:59:13.900323 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900235 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} Oct 30 23:59:13.900323 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900275 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} Oct 30 23:59:13.900323 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900306 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} Oct 30 23:59:13.900462 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900338 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} Oct 30 23:59:13.900462 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.900365 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerStarted","Data":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} Oct 30 23:59:13.901440 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.901420 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8brkm" event={"ID":"a8f07f34-fa12-448b-8054-910fcac344a5","Type":"ContainerStarted","Data":"143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9"} Oct 30 23:59:13.935589 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:13.935539 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.513266234 podStartE2EDuration="4.935521075s" totalImagesPullingTime="3.405463131s" totalInitContainerRuntime="16.79171ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:13.934838167 +0000 UTC m=+66.887633985" watchObservedRunningTime="2026-10-30 23:59:13.935521075 +0000 UTC m=+66.888316892" Oct 30 23:59:14.076477 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:14.076454 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2wlm9"] Oct 30 23:59:14.078784 ip-10-0-136-236 kubenswrapper[2579]: W1030 23:59:14.078756 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02b16bc6_d881_4af8_b248_72a6455b35cd.slice/crio-2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064 WatchSource:0}: Error finding container 2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064: Status 404 returned error can't find the container with id 2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064 Oct 30 23:59:14.418852 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:14.418778 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:14.883295 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:14.883262 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:14.908692 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:14.908654 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wlm9" event={"ID":"02b16bc6-d881-4af8-b248-72a6455b35cd","Type":"ContainerStarted","Data":"2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064"} Oct 30 23:59:15.488617 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:15.488578 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:15.618178 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:15.618099 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:15.915093 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:15.915003 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8brkm" event={"ID":"a8f07f34-fa12-448b-8054-910fcac344a5","Type":"ContainerStarted","Data":"7ee2b062e71e8f84fb37caab04a6e6214f4f1a05983af2afcd99f1db3513032c"} Oct 30 23:59:15.915093 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:15.915073 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8brkm" event={"ID":"a8f07f34-fa12-448b-8054-910fcac344a5","Type":"ContainerStarted","Data":"95490ada15d4528c445158657e50e1259f07fba0904b485c7c9be5dad0317796"} Oct 30 23:59:15.935191 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:15.935137 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8brkm" podStartSLOduration=66.410921148 podStartE2EDuration="1m7.935123615s" totalImagesPullingTime="1.524202467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:15.933804839 +0000 UTC m=+68.886600657" watchObservedRunningTime="2026-10-30 23:59:15.935123615 +0000 UTC m=+68.887919420" Oct 30 23:59:17.622134 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:17.622100 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:17.750435 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:17.750400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:18.223406 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.223323 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:18.344920 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.344879 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:18.790262 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.790228 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:18.929373 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.929339 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2wlm9" event={"ID":"02b16bc6-d881-4af8-b248-72a6455b35cd","Type":"ContainerStarted","Data":"4361e1bdf452e4937217bc3d90cc0306cd6e25a5d9bc4fc641924bb8d35b261c"} Oct 30 23:59:18.929527 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.929428 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:18.947277 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:18.947231 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2wlm9" podStartSLOduration=67.077410071 podStartE2EDuration="1m10.947217861s" totalImagesPullingTime="3.86980779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:18.946584614 +0000 UTC m=+71.899380420" watchObservedRunningTime="2026-10-30 23:59:18.947217861 +0000 UTC m=+71.900013666" Oct 30 23:59:19.463248 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:19.463209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:19.591970 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:19.591927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:20.938061 ip-10-0-136-236 kubenswrapper[2579]: I1030 23:59:20.938025 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqb2v" event={"ID":"dfd6c668-59d3-4840-8ccc-e4e9c7066b44","Type":"ContainerStarted","Data":"7e061688a4a546544ee965b17679891baf352ae2bc70009f6cef25c79f3b3864"} Oct 30 23:59:20.939425 ip-10-0-136-236 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:59:20.954035 ip-10-0-136-236 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:59:20.954200 ip-10-0-136-236 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:59:20.954370 ip-10-0-136-236 systemd[1]: kubelet.service: Consumed 4.403s CPU time. Oct 30 23:59:20.964278 ip-10-0-136-236 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:59:21.036975 ip-10-0-136-236 kubenswrapper[5577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:21.036975 ip-10-0-136-236 kubenswrapper[5577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:59:21.036975 ip-10-0-136-236 kubenswrapper[5577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:21.036975 ip-10-0-136-236 kubenswrapper[5577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:59:21.038980 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038899 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.038980 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038978 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.038980 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038983 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.038980 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038986 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038989 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038992 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038995 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.038998 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039000 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039003 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039005 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039008 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039010 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039014 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039016 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039019 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039022 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039024 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039027 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039030 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039032 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039035 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039038 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.039131 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039041 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039043 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039046 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039049 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039051 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039054 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039056 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039059 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039062 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039064 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039067 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039070 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039072 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039075 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039078 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039081 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039083 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039086 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039089 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.039629 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039091 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039094 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039096 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039099 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039101 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039103 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039106 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039109 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039111 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039113 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039116 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039119 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039121 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039124 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039126 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039130 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039133 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039136 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039138 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039141 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.040164 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039144 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039146 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039149 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039151 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039155 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039159 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039162 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039165 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039168 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039171 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039174 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039176 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039179 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039181 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039183 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039186 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039189 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039191 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039193 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.040679 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039196 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039198 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039201 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039203 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039206 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039208 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039211 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039213 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039216 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039220 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039224 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039228 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039231 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039234 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039237 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039239 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039242 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039245 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039248 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.041172 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039251 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039254 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039256 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039259 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039262 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.039264 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040091 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040098 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040101 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040104 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040107 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040111 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040115 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040119 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040122 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040125 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040127 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040133 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040136 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.041635 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040138 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040141 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040143 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040146 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040149 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040152 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040154 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040157 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040160 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040162 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040165 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040167 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040170 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040173 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040175 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040178 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040181 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040184 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040187 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040189 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.042127 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040192 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040194 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040197 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040199 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040202 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040204 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040207 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040209 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040212 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040214 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040217 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040219 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040223 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040226 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040229 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040233 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040236 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040238 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040241 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.042608 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040244 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040247 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040249 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040252 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040254 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040257 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040259 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040261 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040264 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040266 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040269 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040273 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040275 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040278 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040280 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040283 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040285 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040288 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040290 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.043082 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040293 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040295 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040298 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040301 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040303 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040306 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040308 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040311 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040313 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040317 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040319 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040322 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040324 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040327 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040329 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040331 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040334 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040336 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040339 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.043543 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040341 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040344 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040346 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040349 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040351 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040353 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040356 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040359 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040361 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040364 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040367 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040370 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040372 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040374 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040377 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040379 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040852 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040857 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040860 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040862 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.044041 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040865 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040868 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040871 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040873 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040876 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040879 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040884 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040887 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040890 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040892 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040895 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040897 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040900 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040902 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040905 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040908 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040910 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040913 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040916 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040919 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.044520 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040922 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040924 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040927 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040929 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040932 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040935 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040937 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040940 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040942 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040945 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040947 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040949 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040952 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040955 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040957 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040960 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040963 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040965 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040969 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.045280 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040972 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040974 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040977 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040979 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040982 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040984 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040987 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040991 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040993 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040996 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.040998 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041001 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041004 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041006 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041009 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041011 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041013 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041016 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041018 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041022 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.046037 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041027 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041029 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041032 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041035 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041037 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041040 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041042 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041045 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041048 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041050 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041052 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041056 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041058 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041061 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041064 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041067 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041069 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041072 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041074 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041076 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.046938 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041079 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041081 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041083 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041086 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041088 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041094 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041096 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041099 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041102 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041105 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041108 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041110 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041113 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041115 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041118 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041120 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041122 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041125 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041129 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.047428 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041132 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041134 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041137 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041140 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041229 5577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041236 5577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041242 5577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041249 5577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041256 5577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041261 5577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041268 5577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041274 5577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041277 5577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041281 5577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041284 5577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041287 5577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041290 5577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041293 5577 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041296 5577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041300 5577 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041302 5577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041305 5577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041309 5577 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041312 5577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:59:21.047906 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041315 5577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041318 5577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041321 5577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041326 5577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041329 5577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041332 5577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041335 5577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041340 5577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041343 5577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041346 5577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041349 5577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041352 5577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041358 5577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041361 5577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041364 5577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041367 5577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041370 5577 flags.go:64] FLAG: --enable-server="true" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041372 5577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041376 5577 flags.go:64] FLAG: --event-burst="100" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041379 5577 flags.go:64] FLAG: --event-qps="50" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041382 5577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041385 5577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041388 5577 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041392 5577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:59:21.048498 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041394 5577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041397 5577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041400 5577 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041403 5577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041407 5577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041410 5577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041413 5577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041416 5577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041418 5577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041421 5577 flags.go:64] FLAG: --feature-gates="" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041425 5577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041428 5577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041431 5577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041434 5577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041437 5577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041441 5577 flags.go:64] FLAG: --help="false" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041444 5577 flags.go:64] FLAG: --hostname-override="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041447 5577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041450 5577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041453 5577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041457 5577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041461 5577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041463 5577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:59:21.049106 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041466 5577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041469 5577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041472 5577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041475 5577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041478 5577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041481 5577 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041484 5577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041486 5577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041489 5577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041492 5577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041495 5577 flags.go:64] FLAG: --lock-file="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041497 5577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041500 5577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041503 5577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041509 5577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041512 5577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041515 5577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041517 5577 flags.go:64] FLAG: --logging-format="text" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041520 5577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041523 5577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041526 5577 flags.go:64] FLAG: --manifest-url="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041529 5577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041533 5577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041536 5577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041541 5577 flags.go:64] FLAG: --max-pods="110" Oct 30 23:59:21.049648 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041543 5577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041546 5577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041549 5577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041552 5577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041556 5577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041559 5577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041562 5577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041569 5577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041572 5577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041575 5577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041578 5577 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041581 5577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041584 5577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041587 5577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041590 5577 flags.go:64] FLAG: --port="10250" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041593 5577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041596 5577 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0d378a1458ba62f5e" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041599 5577 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041601 5577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041604 5577 flags.go:64] FLAG: --register-node="true" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041607 5577 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041610 5577 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041614 5577 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041617 5577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041620 5577 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041624 5577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:59:21.050269 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041627 5577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041630 5577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041633 5577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041636 5577 flags.go:64] FLAG: --runonce="false" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041639 5577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041643 5577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041646 5577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041649 5577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041652 5577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041655 5577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041659 5577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041662 5577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041665 5577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041668 5577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041670 5577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041673 5577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041676 5577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041679 5577 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041682 5577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041687 5577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041689 5577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041692 5577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041696 5577 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041699 5577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041702 5577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:59:21.050900 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041719 5577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041722 5577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041725 5577 flags.go:64] FLAG: --v="2" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041730 5577 flags.go:64] FLAG: --version="false" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041735 5577 flags.go:64] FLAG: --vmodule="" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041738 5577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.041741 5577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041869 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041872 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041875 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041878 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041880 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041884 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041886 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041889 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041891 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041893 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041896 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041898 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041901 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041903 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.051521 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041906 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041908 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041910 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041913 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041915 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041918 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041920 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041922 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041925 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041928 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041930 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041932 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041935 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041937 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041940 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041942 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041945 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041947 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041949 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041952 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.052068 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041954 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041956 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041959 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041962 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041965 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041967 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041970 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041972 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041975 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041977 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041980 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041982 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.041999 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042003 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042007 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042010 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042013 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042015 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042017 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042020 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.052550 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042022 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042024 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042027 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042029 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042032 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042035 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042039 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042042 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042045 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042048 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042051 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042054 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042057 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042060 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042064 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042068 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042071 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042074 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042077 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.053124 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042079 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042082 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042085 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042087 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042089 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042092 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042095 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042097 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042099 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042103 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042107 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042110 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042114 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042117 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042119 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042122 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042125 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042127 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042129 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.053680 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042132 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042135 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042137 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042140 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042143 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042146 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042148 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042151 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042155 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042158 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042161 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042163 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042166 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042168 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.042195 5577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042231 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.054234 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042235 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042239 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042242 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042244 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042247 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042249 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042252 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042255 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042257 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042260 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042262 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042265 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042268 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042270 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042273 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042275 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042278 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042280 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042283 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042285 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.054619 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042287 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042290 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042293 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042295 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042298 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042302 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042304 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042307 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042309 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042312 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042314 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042317 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042319 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042322 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042324 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042327 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042329 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042332 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042334 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.055119 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042336 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042339 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042341 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042344 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042346 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042349 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042351 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042354 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042356 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042359 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042361 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042364 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042366 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042369 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042371 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042374 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042377 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042379 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042383 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042385 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.055634 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042388 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042390 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042393 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042395 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042397 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042400 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042402 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042405 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042407 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042411 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042415 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042418 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042420 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042423 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042425 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042428 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042430 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042433 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042435 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.056276 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042438 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042440 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042442 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042445 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042447 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042450 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042452 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042455 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042457 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042460 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042463 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042466 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042469 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042472 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042476 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042479 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042482 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042485 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042487 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.056784 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042490 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042492 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042495 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042497 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042500 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042502 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042505 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.042507 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048039 5577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048055 5577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048073 5577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048077 5577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048136 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048141 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048144 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048147 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048150 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048153 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048155 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048158 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.057235 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048160 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048163 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048165 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048168 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048171 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048174 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048176 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048179 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048181 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048184 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048186 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048189 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048192 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048194 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048197 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048199 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048202 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048204 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048207 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048209 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.057759 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048215 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048217 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048220 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048223 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048225 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048228 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048230 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048233 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048235 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048238 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048241 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048243 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048246 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048248 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048265 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048268 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048272 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048275 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048281 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048284 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.058230 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048287 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048289 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048292 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048294 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048297 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048300 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048302 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048305 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048307 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048310 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048312 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048315 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048317 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048319 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048322 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048324 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048327 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048330 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048332 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.058703 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048335 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048337 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048340 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048342 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048345 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048347 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048349 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048352 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048354 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048357 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048359 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048362 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048365 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048367 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048370 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048372 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048375 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048377 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048380 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048382 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.059167 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048385 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048387 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048391 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048395 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048398 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048401 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048403 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048406 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048408 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048410 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048414 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048418 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048422 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048425 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048427 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048430 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048432 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048435 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.059638 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048438 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048466 5577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048604 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048608 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048612 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048614 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048617 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048620 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048623 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048625 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048628 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048631 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048633 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048636 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048638 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048641 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.060104 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048643 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048646 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048648 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048651 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048653 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048656 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048658 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048661 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048666 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048669 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048671 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048674 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048677 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048679 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048682 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048684 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048686 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048689 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048691 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048694 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.060478 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048696 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048699 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048702 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048721 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048726 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048728 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048731 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048734 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048736 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048739 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048741 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048744 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048746 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048749 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048752 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048754 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048756 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048760 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048763 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.061263 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048766 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048771 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048774 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048777 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048779 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048782 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048784 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048787 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048789 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048792 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048794 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048797 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048799 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048802 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048804 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048807 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048809 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048812 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048815 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048817 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.062107 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048820 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048822 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048825 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048827 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048831 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048835 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048838 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048841 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048843 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048846 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048848 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048850 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048853 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048858 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048860 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048863 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048865 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048868 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048870 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.063003 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048873 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048875 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048878 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048880 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048883 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048886 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048888 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048890 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048893 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048895 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048898 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048900 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048903 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048906 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.048929 5577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:59:21.063842 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048962 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048968 5577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048972 5577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048975 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048978 5577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048981 5577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048984 5577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048986 5577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048989 5577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048991 5577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048994 5577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.048997 5577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049003 5577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049005 5577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049008 5577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049010 5577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049013 5577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049015 5577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049018 5577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:59:21.064585 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049020 5577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049023 5577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049030 5577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049033 5577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049035 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049038 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049041 5577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049043 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049046 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049048 5577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049050 5577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049054 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049056 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049059 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049061 5577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049064 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049066 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049068 5577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049071 5577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049073 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:59:21.065475 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049076 5577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049079 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049081 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049084 5577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049088 5577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049091 5577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049094 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049096 5577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049099 5577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049102 5577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049104 5577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049107 5577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049110 5577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049113 5577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049115 5577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049118 5577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049121 5577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049123 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049126 5577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:59:21.066411 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049128 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049131 5577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049133 5577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049136 5577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049138 5577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049141 5577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049144 5577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049147 5577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049149 5577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049152 5577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049154 5577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049156 5577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049159 5577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049161 5577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049164 5577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049166 5577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049169 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049172 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049175 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049178 5577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:59:21.067492 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049180 5577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049183 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049185 5577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049189 5577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049192 5577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049195 5577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049198 5577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049200 5577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049203 5577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049205 5577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049208 5577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049210 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049212 5577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049215 5577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049217 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049220 5577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049222 5577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049225 5577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049228 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:59:21.068460 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049232 5577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049234 5577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049236 5577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049239 5577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049241 5577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049244 5577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049246 5577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049248 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:21.049251 5577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.049303 5577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050251 5577 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050306 5577 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050481 5577 server.go:1014] "Starting client certificate rotation" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050620 5577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050668 5577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:53:08 +0000 UTC" deadline="2028-03-26 21:30:54.68983033 +0000 UTC" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050695 5577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12309h31m33.6391375s" Oct 30 23:59:21.069296 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.050916 5577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:59:21.070143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.052121 5577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:59:21.070143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.053591 5577 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:59:21.070143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.055305 5577 log.go:25] "Validated CRI v1 image API" Oct 30 23:59:21.070143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.056068 5577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:59:21.070143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.058122 5577 fs.go:135] Filesystem UUIDs: map[48e9417f-a2ad-4e02-bfbd-0fa5f0fa70ed:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 d7696fec-362f-417b-8746-0141c5787a33:/dev/nvme0n1p4] Oct 30 23:59:21.070391 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.058135 5577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064/userdata/shm major:0 minor:422 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7/userdata/shm major:0 minor:284 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af/userdata/shm major:0 minor:310 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02b16bc6-d881-4af8-b248-72a6455b35cd/volumes/kubernetes.io~projected/kube-api-access-m4598:{mountpoint:/var/lib/kubelet/pods/02b16bc6-d881-4af8-b248-72a6455b35cd/volumes/kubernetes.io~projected/kube-api-access-m4598 major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/alertmanager/9 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/config-reloader/6 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~empty-dir/config-out major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/kube-api-access-767n5:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/kube-api-access-767n5 major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/tls-assets major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/config-volume major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/web-config major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/118f5c91-d3a9-434b-b931-89be645e00e2/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/118f5c91-d3a9-434b-b931-89be645e00e2/volumes/kubernetes.io~secret/agent-certs major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2026287d-f8ba-4ba5-9681-d6d74a7d1058/volumes/kubernetes.io~projected/kube-api-access-vjlgk:{mountpoint:/var/lib/kubelet/pods/2026287d-f8ba-4ba5-9681-d6d74a7d1058/volumes/kubernetes.io~projected/kube-api-access-vjlgk major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/36bdf75e-c844-4a33-b5dd-ac33878a0186/volumes/kubernetes.io~projected/kube-api-access-6pggm:{mountpoint:/var/lib/kubelet/pods/36bdf75e-c844-4a33-b5dd-ac33878a0186/volumes/kubernetes.io~projected/kube-api-access-6pggm major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~projected/kube-api-access-65bn7:{mountpoint:/var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~projected/kube-api-access-65bn7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:200 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~projected/kube-api-access-npvjn:{mountpoint:/var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~projected/kube-api-access-npvjn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~secret/cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~projected/kube-api-access-6xb7l:{mountpoint:/var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~projected/kube-api-access-6xb7l major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~secret/metrics-tls major:0 minor:196 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~projected/kube-api-access-ndwrv:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~projected/kube-api-access-ndwrv major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:274 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bd67fa-8a46-4098-a7b2-0dab23bf0da1/volumes/kubernetes.io~projected/kube-api-access-mlx47:{mountpoint:/var/lib/kubelet/pods/85bd67fa-8a46-4098-a7b2-0dab23bf0da1/volumes/kubernetes.io~projected/kube-api-access-mlx47 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~projected/kube-api-access-hcmj5:{mountpoint:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~projected/kube-api-access-hcmj5 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~projected/kube-api-access-6fv6v:{mountpoint:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~projected/kube-api-access-6fv6v major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/config-reloader/2 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/prometheus/15 major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~empty-dir/config-out major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/kube-api-access-8pgql:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/kube-api-access-8pgql major:0 minor:340 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/tls-assets major:0 minor:326 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/config major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:332 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:333 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/web-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~projected/kube-api-access-tbtk9:{mountpoint:/var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~projected/kube-api-access-tbtk9 major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~secret/metrics-certs major:0 minor:379 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4382ac5-fea6-4cf2-b1a6-c06514792224/volumes/kubernetes.io~projected/kube-api-access-hdxzw:{mountpoint:/var/lib/kubelet/pods/b4382ac5-fea6-4cf2-b1a6-c06514792224/volumes/kubernetes.io~projected/kube-api-access-hdxzw major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c869979e-11d8-4619-8c86-04d2b4acedd6/volumes/kubernetes.io~projected/kube-api-access-ts78k:{mountpoint:/var/lib/kubelet/pods/c869979e-11d8-4619-8c86-04d2b4acedd6/volumes/kubernetes.io~projected/kube-api-access-ts78k major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccc77441-8a6d-4a2a-95ca-553f8870b21c/volumes/kubernetes.io~projected/kube-api-access-mq2wt:{mountpoint:/var/lib/kubelet/pods/ccc77441-8a6d-4a2a-95ca-553f8870b21c/volumes/kubernetes.io~projected/kube-api-access-mq2wt major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfd6c668-59d3-4840-8ccc-e4e9c7066b44/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/dfd6c668-59d3-4840-8ccc-e4e9c7066b44/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~projected/kube-api-access-wmw27:{mountpoint:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~projected/kube-api-access-wmw27 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~projected/kube-api-access-d62jc:{mountpoint:/var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~projected/kube-api-access-d62jc major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:305 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/c427df237cd6e0338e48b4d1d7cd9bc045c8badc349d28690d9aae22842fae02/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f0dfc8597a1f599172acc27764aa6e347602980627da6ab0e9f8c3149aae1d1c/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/705acc6d7419cee3f60b24d39095428d85856b16ef94e5ab41e4ba4ee21f7e44/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/8989872da76718b710e470ff86298776e352357752e80b2a494ebd202ed17faa/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/3d0cd7e88c92e794799cf6fc20d0adec42a657314627045f68b33334be1e180a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bb59afc018d0c4a6c13107ae992b3d3f85b186e4e67006116738f9307396e643/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/1a65d24c9e7918419ff31c748d069fc3a98ae60b106e3b71c30e72573c3b8741/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c8b0722843742e36c1fc44b0088b09f41e47586056b49b2f219c8d58c3beaacc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/44100ca278da33b3d1dcbce116e50f053d6189501869c9afa5785c256bb4ec19/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/e4c465c2a18a2f45abbf8e60e657d493ae2e90bcf6f6a83c758b2fa7913c9d85/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/37e4b62e27fc7cd4d7b3ab6c33f14261d52dad381e9f1986973d5dd757205124/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/f4b5544042c3f7351a0d3c3695fa879a2016894ed47777fb6ec06b47f2c82eb0/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6877486da0d98e280a02dccc56691c34140957ab1d89f8d46bf4fc620775e71f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-161:{mountpoint:/var/lib/containers/storage/overlay/c17b907fe45c2d6f30b9a0c4d15955125ea320eb431bfc5e7958a19753830508/merged major:0 minor:161 fsType:overlay blockSize:0} overlay_0-166:{mountpoint:/var/lib/containers/storage/overlay/4f602f71bf11fc92fa4f91581ab2235be043d78b55adeb995b75371e303eac10/merged major:0 minor:166 fsType:overlay blockSize:0} overlay_0-171:{mountpoint:/var/lib/containers/storage/overlay/24d84b27bc2980cb2cf7bdc69100ce889e1c2405e4eb836dffd9befa53ce2f9d/merged major:0 minor:171 fsType:overlay blockSize:0} overlay_0-176:{mountpoint:/var/lib/containers/storage/overlay/5087f236b0f663efdeaa5c5bf616c018216e345329f570e8493441e396eb51fe/merged major:0 minor:176 fsType:overlay blockSize:0} overlay_0-181:{mountpoint:/var/lib/containers/storage/overlay/c5bfea4068267ef47c61d174372f3fb5f34b1ce4c602868b2846be94042a4a87/merged major:0 minor:181 fsType:overlay blockSize:0} overlay_0-186:{mountpoint:/var/lib/containers/storage/overlay/b17c4713d93c5885ea67c0e360bd713c3838e448521c590b4ed59b7765a2018f/merged major:0 minor:186 fsType:overlay blockSize:0} overlay_0-191:{mountpoint:/var/lib/containers/storage/overlay/73f599a0e891283657c1587c6979268ba3702d3b1da75b2b560eb5ae5752d558/merged major:0 minor:191 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/f56669577179a27ce9bcec565b61391f9a0e7121f750cd192cebf28a28b0eadd/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/1f00fca97add133c0d421cde30ba9d9d6d7e89cfe24a62a6ade6761eeca3a44a/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c3c73a9a0265ab63366f174d859576601a3d02c99e107a08a578c4f13340c468/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ba9ae5b73e86188ae69ea22e8ce68042103412b28ffe1a9794eec87dfb9c193a/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/d5941436e7a3f515535184a127b084e7514572fec6b6d4c5cb84691ff1e4809b/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/e0d661caed279087ae0e6b95a3dde9641b7836807efad24174cfbfb2c3a9bd88/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/b59d57da762d4bd1a4715797188afd3b635c9db3751892e01daa988bf39bad0b/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/71542c265992872a881c213bcaa604ba5364f2c70a67aac6e26bc52b3a277552/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/05db5769311e4e3952715e056b409d7ac455f5045c5f8638d8225c0f38b4a40f/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/a258e096bab3fd4e3cc566f2aa4289964fd0c08634dbe25368d1a38591aa5ba1/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/c042a66cc05be44fcefa999d872e294fc1fc9bd14d4bdcd75a72855c9918eab1/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/831110ca722b6aab95a741d0bcd5ad2819ce4bb0bad91d724418c6ef39a1033c/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/a0f5f91433650ccae641e5efd192ec46dc49939b19b9e82102509b4bf616c5ec/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/63d7e36a59e8b87ad67c237426eb3ebc90077489ee9ade2e97ebabe207ca7f50/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/5e6e68d91229f5411e1a8f3a899ebcc1fbf31917d39aaebff57d885d9a8bfb51/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/322235001e6726e1b27f23b269e98b17c31f18fb71d676071f20c9ef31a118f2/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/2a03eadbe27eb01cda32d4fe18f66307dc119ac135da958c90df5b26f5328d73/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/b6702f20fe58399c150bf2df33d354a02755f67455d3c1885764ff314090d4ef/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/c2a1456e28e0b5ccf5b69f7b59f41e44bdf65d74a916788068cb3df701b152ba/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/80a1163d8f790380716e2157aeca4f6d7b5a7a1dae24ccc6aa46f67a5906b203/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/456a3cba06c056aee76e45892e5aaeae06923b4fe6cf695deef158141a509c74/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/4a78fa17e2b54815b617a560fad1563579af54260ef140afbc85311accfcad2e/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/7908d2df206a49d05a2069492691524ebaf678c4d4df5f3df4c0c235db5ec75f/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-351:{mountpoint:/var/lib/containers/storage/overlay/0651332e2229bdf1f1059450a16a3f1cb7cc6f2628245e6395a363d7753ce9cb/merged major:0 minor:351 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/c615d2a1edc3adfbe0f34473fcc01c3c5b1fb39cbc846a4d154277cc4c689a71/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/10eb49fa95333ece578eef029bfe59b22168caf4e8446ceb13dc2ce915c7f137/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-364:{mountpoint:/var/lib/containers/storage/overlay/de3ae8807d2b1c4a01602f3f0cc6f330e1b27472ce03d443d5e7467a10d8bd0a/merged major:0 minor:364 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/437e63027eff9e3c5f94ae970359d185a595b352ceab85e2457510d68046c96a/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/04fb2b409e711665850ce8b2474fce9cef3215748cdd737a8a8ba8508e664ff5/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/323e4a75df3a2238578a85d96d33b5b533dd6d5347f9b4651c5e6dc7d4ff229c/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-386:{mountpoint:/var/lib/containers/storage/overlay/79389ea28af824df8e3022bc84e4479be3d35cc82726c424406d23b12348261d/merged major:0 minor:386 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/87f795948d871fa91c5b03ba6379a655d93cc602783f05cf7b534ee9fd3e2814/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/90573afb8a223d7bce6939f03df5bda5e0e0bbb339092439866297b532de506c/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/b43f571bf091a039e1151741bdf240ebd4d7496d85d652f4a340621f7569d241/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/53059b5fce1ad112e7c9e1d57778ea8dd6f2fc46df735d45089e287722e48e24/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/6d7bb55611f04b5d9804c68af58bb60b8f9188e07d01aa522eba990648683468/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/0f2a2c3fa1b7aff07e338feaac6b90456d0b7dff477c58387512173d623b03a0/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/b77ed4dedbcf3dd494ba0afb3adc9a8a8823ff77617442abf513058c156218e0/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/652ae4b58ef5b1d60a0e1e3ff908d248c8dfd5b4b5fefc2542fa0bbca069ee09/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-430:{mountpoint:/var/lib/containers/storage/overlay/97aa69002d29150ee03e90477334658ef7e5302a33a7264d06908346e3de1abb/merged major:0 minor:430 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/d022ea679f72bcb1f08f16af5b159518c3aa28d332e8c712b9805a3d7ab32e09/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-441:{mountpoint:/var/lib/containers/storage/overlay/e2ee252d3a791d5244ffbd42c5e520f077badb9ff79a377aa869179f0bde7f65/merged major:0 minor:441 fsType:overlay blockSize:0} overlay_0-45:{mountpoint:/var/lib/containers/storage/overlay/316fd1b182ce1931377434110ba57ee26307e6ec161b3becda589ca68de6ab91/merged major:0 minor:45 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0e71aba22697d074a62a7c22210b631d40b596457a78978478291bec84de44c0/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bef0dbf3bd3bdc4142b793345ec872a21716f3f303b5adfadb5129c92cd4179a/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/2802eafc77b196d5d929638c4f073757befddd37494b3dfe450f09b538e1473b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/318d0693db4926b7772c143fb28a0e91bb91e3ba342fb2677f4e78f69f545842/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/c4dbeb1ec2c5069a230914fa77dd952720557f235ea095ead62424c56ff388ae/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/d07328bf7db100f01974c4474b4672ed2c321b117fba23a27ebd5d8b8d37094c/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/0494ff3358a96ab53d67bf1e573ebcdd128cebdccd802fd1d098488ebd14506e/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/388ea3aa69c9b6f5fb402f82626d56e5fa777b088acd8c1eef335298a8a97072/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/2f45e340bfd0deb58031a654e94dd1a66152f6cdd91994014598b6c60cb05257/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a31d1cb0d5d8b6acf947cf041d33bfb6eed7dc4ce676355ccc30a7aca2f1eebd/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 23:59:21.070856 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.069564 5577 manager.go:217] Machine: {Timestamp:2026-10-30 23:59:21.068751812 +0000 UTC m=+0.060411396 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3078151 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22c51afc528781a6d66259fe50a019 SystemUUID:ec22c51a-fc52-8781-a6d6-6259fe50a019 BootID:25cee24e-ac78-44ae-b877-afcf3927b0f4 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c34048c88cf82ac68bf494afd397cc17992452696ef74c995c034c6d3b57b1a/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8525411927f93aac7c72a23a88bdf36b54941a2c73c076bc8a7058ffd7e0301/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volumes/kubernetes.io~projected/kube-api-access-hcmj5 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~projected/kube-api-access-npvjn DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:326 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:379 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bd67fa-8a46-4098-a7b2-0dab23bf0da1/volumes/kubernetes.io~projected/kube-api-access-mlx47 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c59a62dd7afe510bc1c4645d8ec295087f020113d03ebfac0cbda2f07cc52f83/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-161 DeviceMajor:0 DeviceMinor:161 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~projected/kube-api-access-6xb7l DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:259 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:332 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:339 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ccc77441-8a6d-4a2a-95ca-553f8870b21c/volumes/kubernetes.io~projected/kube-api-access-mq2wt DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d23f94afe7c5d4a895ba2bec868954b9bca094cc4d04820ab9c686ce88999f0c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b198eece347029ccaa377d0f6d55a4df874eff8e7f5dc77a0b474a6637e58b2e/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df1034813317384654320c41a47701cea9e2c55c3cb2e9786b8077889f3f17af/userdata/shm DeviceMajor:0 DeviceMinor:310 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:265 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/575c78f4109bb7e2433b98f60cec8ce6cc7bc4f6e1b07b2e2ff7e5b8c4e99ad2/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:260 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d748726b9186f73380a9daf452d3b25334e6eb3872661f2d4da08f697294064/userdata/shm DeviceMajor:0 DeviceMinor:422 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-191 DeviceMajor:0 DeviceMinor:191 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a28c09406df4913949492654dfea06d0a9c27ab8f78c4042feea1a34a136797b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:305 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/143caa36249a8caa9d1ba27165aaa4d63570ec5ed9a4ffcc5fafd4c6ab368bd9/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-181 DeviceMajor:0 DeviceMinor:181 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:280 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:281 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:335 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5c12cca1-08f4-49f6-a54d-a02e676f6b54/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:278 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:262 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-441 DeviceMajor:0 DeviceMinor:441 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/36bdf75e-c844-4a33-b5dd-ac33878a0186/volumes/kubernetes.io~projected/kube-api-access-6pggm DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:330 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f70e9c8a3332d40a3ea6648e4d2152cf1f1d36782a979f875a92fde377dd588f/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/99dfb869-c9bf-41d2-832a-d69ae57a802b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8f07f34-fa12-448b-8054-910fcac344a5/volumes/kubernetes.io~projected/kube-api-access-tbtk9 DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~projected/kube-api-access-wmw27 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:331 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-364 DeviceMajor:0 DeviceMinor:364 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-430 DeviceMajor:0 DeviceMinor:430 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-176 DeviceMajor:0 DeviceMinor:176 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~projected/kube-api-access-ndwrv DeviceMajor:0 DeviceMinor:283 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~projected/kube-api-access-8pgql DeviceMajor:0 DeviceMinor:340 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-386 DeviceMajor:0 DeviceMinor:386 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-45 DeviceMajor:0 DeviceMinor:45 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74c9c8bd4d435d1b89c5fe633144e015744a365686c90443838b1d979ca8bcaf/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:334 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-166 DeviceMajor:0 DeviceMinor:166 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:251 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b4382ac5-fea6-4cf2-b1a6-c06514792224/volumes/kubernetes.io~projected/kube-api-access-hdxzw DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c7657a9be9d0b2a2512076a868962c2575fefc1feb47f5aa3576912ae474d4d7/userdata/shm DeviceMajor:0 DeviceMinor:284 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:333 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:337 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/118f5c91-d3a9-434b-b931-89be645e00e2/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-171 DeviceMajor:0 DeviceMinor:171 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-186 DeviceMajor:0 DeviceMinor:186 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~projected/kube-api-access-65bn7 DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~projected/kube-api-access-767n5 DeviceMajor:0 DeviceMinor:266 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ec194b59-61ad-4c7f-b93c-1417107e0269/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:250 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:261 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfd6c668-59d3-4840-8ccc-e4e9c7066b44/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:383 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17d740bd39a772e63b7324bd11c2d2793c8c118a73e1258949d7d74a989258e7/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe424bc5-4967-481f-bc33-3f7b3db28a9e/volumes/kubernetes.io~projected/kube-api-access-d62jc DeviceMajor:0 DeviceMinor:309 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~projected/kube-api-access-6fv6v DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:264 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:336 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5b456bf9-fa54-4740-998e-d0788f096c3c/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:200 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:274 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6aeaa875-fa12-46cc-b25d-453ce96d4a1e/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:279 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2bae7382291794ed4ca87114e82d73ce06aef5e4ad9922b30c357bc0f5caf56/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/692f415f-94bd-4fd2-ba6c-468145504bc0/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:196 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d78e614b8f742dd5e3cebbf6c69764faa0088897f61b306251012ae48fcbe78b/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9ac9fb68-1982-42a6-8a34-7b9fa61a7565/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:263 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-351 DeviceMajor:0 DeviceMinor:351 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2026287d-f8ba-4ba5-9681-d6d74a7d1058/volumes/kubernetes.io~projected/kube-api-access-vjlgk DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72b65ee627891ccdab8b0e37c91a56bdc51639cb4178105fec0628b9a01dfba2/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:261 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:338 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/02b16bc6-d881-4af8-b248-72a6455b35cd/volumes/kubernetes.io~projected/kube-api-access-m4598 DeviceMajor:0 DeviceMinor:399 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c869979e-11d8-4619-8c86-04d2b4acedd6/volumes/kubernetes.io~projected/kube-api-access-ts78k DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:143caa36249a8ca MacAddress:9a:7d:c3:1c:aa:9c Speed:10000 Mtu:8901} {Name:2d748726b9186f7 MacAddress:b6:8a:02:ab:7f:a4 Speed:10000 Mtu:8901} {Name:4324e86af798960 MacAddress:86:b7:e3:08:eb:de Speed:10000 Mtu:8901} {Name:72b65ee627891cc MacAddress:a6:f5:a6:95:3d:91 Speed:10000 Mtu:8901} {Name:a28c09406df4913 MacAddress:6e:c0:c2:c3:c3:3b Speed:10000 Mtu:8901} {Name:a2bae7382291794 MacAddress:66:01:ec:e9:8d:3d Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:35:15:cb Speed:0 Mtu:9001} {Name:br-int MacAddress:6e:64:20:15:99:9f Speed:0 Mtu:8901} {Name:c7657a9be9d0b2a MacAddress:e6:1e:32:1d:a4:d8 Speed:10000 Mtu:8901} {Name:d2622315834f4b2 MacAddress:b6:bc:07:9a:4c:87 Speed:10000 Mtu:8901} {Name:d78e614b8f742dd MacAddress:b2:c6:b8:68:3c:30 Speed:10000 Mtu:8901} {Name:df1034813317384 MacAddress:d6:03:70:f3:86:51 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:35:15:cb Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ce:6c:cc:37:af:14 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:12:ee:dd:01:44:2f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:59:21.070856 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070289 5577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:59:21.070856 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070352 5577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:59:21.070856 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070662 5577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070686 5577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-236.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070884 5577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070896 5577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070911 5577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.070937 5577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071043 5577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:59:21.071304 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071252 5577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071385 5577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071429 5577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071443 5577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071457 5577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071470 5577 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:59:21.071608 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.071481 5577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:59:21.072281 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.072265 5577 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:59:21.072281 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.072280 5577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:59:21.072783 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.072750 5577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:59:21.075741 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.075726 5577 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:59:21.075741 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.075739 5577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:59:21.075867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.075857 5577 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:59:21.076064 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076053 5577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:59:21.076095 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076082 5577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:59:21.076177 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076168 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076183 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076191 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076197 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076204 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076213 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:59:21.076218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076220 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076230 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076248 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076257 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076268 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076282 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076304 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:59:21.076422 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.076310 5577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:59:21.079952 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.079938 5577 server.go:1263] "Started kubelet" Oct 30 23:59:21.080060 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.080029 5577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:59:21.080592 ip-10-0-136-236 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:59:21.081828 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.080617 5577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:59:21.081828 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.080747 5577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:59:21.081828 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.081058 5577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:59:21.085800 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.085769 5577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:59:21.087090 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.087069 5577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:59:21.087652 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.087632 5577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:59:21.087652 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.087652 5577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:59:21.087911 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.087752 5577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:59:21.088347 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.088299 5577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-236.us-east-2.compute.internal\" not found" Oct 30 23:59:21.088448 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.088425 5577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:59:21.088535 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.088496 5577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:53:10 +0000 UTC" deadline="2028-06-24 21:13:57.595897327 +0000 UTC" Oct 30 23:59:21.088599 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.088537 5577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14469h14m36.507364157s" Oct 30 23:59:21.090865 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090818 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02b16bc6-d881-4af8-b248-72a6455b35cd" volumeName="kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090884 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99dfb869-c9bf-41d2-832a-d69ae57a802b" volumeName="kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090899 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090914 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090925 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c12cca1-08f4-49f6-a54d-a02e676f6b54" volumeName="kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090937 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99dfb869-c9bf-41d2-832a-d69ae57a802b" volumeName="kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090951 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090963 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 23:59:21.090973 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090976 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090988 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.090999 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091009 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4382ac5-fea6-4cf2-b1a6-c06514792224" volumeName="kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091023 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99dfb869-c9bf-41d2-832a-d69ae57a802b" volumeName="kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091034 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091050 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091064 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36bdf75e-c844-4a33-b5dd-ac33878a0186" volumeName="kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091085 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36bdf75e-c844-4a33-b5dd-ac33878a0186" volumeName="kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091098 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36bdf75e-c844-4a33-b5dd-ac33878a0186" volumeName="kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091109 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f415f-94bd-4fd2-ba6c-468145504bc0" volumeName="kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091121 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091144 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118f5c91-d3a9-434b-b931-89be645e00e2" volumeName="kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091155 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f415f-94bd-4fd2-ba6c-468145504bc0" volumeName="kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091166 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091178 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091190 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091203 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8f07f34-fa12-448b-8054-910fcac344a5" volumeName="kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091214 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4382ac5-fea6-4cf2-b1a6-c06514792224" volumeName="kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091225 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec194b59-61ad-4c7f-b93c-1417107e0269" volumeName="kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091245 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091257 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091273 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091293 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091301 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091308 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8f07f34-fa12-448b-8054-910fcac344a5" volumeName="kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091316 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc77441-8a6d-4a2a-95ca-553f8870b21c" volumeName="kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091325 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec194b59-61ad-4c7f-b93c-1417107e0269" volumeName="kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091337 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091349 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b456bf9-fa54-4740-998e-d0788f096c3c" volumeName="kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091356 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f415f-94bd-4fd2-ba6c-468145504bc0" volumeName="kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091366 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091374 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bd67fa-8a46-4098-a7b2-0dab23bf0da1" volumeName="kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091383 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 23:59:21.091377 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091393 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfd6c668-59d3-4840-8ccc-e4e9c7066b44" volumeName="kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091405 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091416 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091424 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="118f5c91-d3a9-434b-b931-89be645e00e2" volumeName="kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091431 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b456bf9-fa54-4740-998e-d0788f096c3c" volumeName="kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091439 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091448 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091455 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091462 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091469 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091481 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091489 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091496 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091504 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091516 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c869979e-11d8-4619-8c86-04d2b4acedd6" volumeName="kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091523 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe424bc5-4967-481f-bc33-3f7b3db28a9e" volumeName="kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091530 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091539 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5c12cca1-08f4-49f6-a54d-a02e676f6b54" volumeName="kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091551 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091566 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091578 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091593 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091606 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091618 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2026287d-f8ba-4ba5-9681-d6d74a7d1058" volumeName="kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091631 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2026287d-f8ba-4ba5-9681-d6d74a7d1058" volumeName="kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091642 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99dfb869-c9bf-41d2-832a-d69ae57a802b" volumeName="kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091654 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091668 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2026287d-f8ba-4ba5-9681-d6d74a7d1058" volumeName="kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091680 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="36bdf75e-c844-4a33-b5dd-ac33878a0186" volumeName="kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091691 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b456bf9-fa54-4740-998e-d0788f096c3c" volumeName="kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091703 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091736 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091748 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccc77441-8a6d-4a2a-95ca-553f8870b21c" volumeName="kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091764 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b456bf9-fa54-4740-998e-d0788f096c3c" volumeName="kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091777 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="692f415f-94bd-4fd2-ba6c-468145504bc0" volumeName="kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091788 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bd67fa-8a46-4098-a7b2-0dab23bf0da1" volumeName="kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091802 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" volumeName="kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091815 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6948ba0-e600-4cfe-8e6d-e8e912da4861" volumeName="kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091829 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe424bc5-4967-481f-bc33-3f7b3db28a9e" volumeName="kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091843 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091855 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091867 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="99dfb869-c9bf-41d2-832a-d69ae57a802b" volumeName="kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091884 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091906 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" volumeName="kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091917 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6aeaa875-fa12-46cc-b25d-453ce96d4a1e" volumeName="kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091928 5577 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec194b59-61ad-4c7f-b93c-1417107e0269" volumeName="kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp" seLinuxMountContext="" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091941 5577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.091950 5577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092066 5577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092333 5577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092385 5577 factory.go:55] Registering systemd factory Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092404 5577 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092767 5577 factory.go:153] Registering CRI-O factory Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092785 5577 factory.go:223] Registration of the crio container factory successfully Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092897 5577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092912 5577 factory.go:103] Registering Raw factory Oct 30 23:59:21.092949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.092922 5577 manager.go:1199] Started watching for new ooms in manager Oct 30 23:59:21.095595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.093263 5577 manager.go:319] Starting recovery of all containers Oct 30 23:59:21.100777 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.098736 5577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:21.100777 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.099288 5577 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:59:21.114689 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.114646 5577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:59:21.120566 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.120541 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:59:21.122413 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.122395 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:59:21.123438 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.123420 5577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:59:21.123547 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.123455 5577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:59:21.123547 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.123479 5577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:59:21.123547 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.123521 5577 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:59:21.128465 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.128019 5577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:59:21.132880 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.132855 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" exitCode=0 Oct 30 23:59:21.139440 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.139417 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" exitCode=0 Oct 30 23:59:21.143548 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.143523 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 30 23:59:21.144005 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.143979 5577 generic.go:356] "Generic (PLEG): container finished" podID="99dfb869-c9bf-41d2-832a-d69ae57a802b" containerID="f544c44d1d90495b164b0e62fb31a120e88341d8291e05136fb77d616c45f80c" exitCode=1 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152090 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="62dfe26f7587906c427faeefd1fa19909ce0115de036cb84863d1b8d9ca6292d" exitCode=0 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152119 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="54f61acdd5c7af30098f64284b29f25a5ae670fe846814ac81d169f4eabee118" exitCode=0 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152131 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="d30ed5c5327c9eac96479b0b48f9a7a57710b69d5a6ad4b0393389e1f87e5597" exitCode=0 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152142 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="afee42a100231180a0b1dfa714ad882b16493c21a0cac9627233491ea96cbdf4" exitCode=0 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152154 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="8aa1e6d28566d17924c42400e88b827bc3051e1b285d0c27b651e9fa385ef423" exitCode=0 Oct 30 23:59:21.152280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.152173 5577 generic.go:356] "Generic (PLEG): container finished" podID="36bdf75e-c844-4a33-b5dd-ac33878a0186" containerID="c6313539a576676bff78cae57898da0693bed7ad6c1cde25b76be924574e2137" exitCode=0 Oct 30 23:59:21.167646 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.167621 5577 generic.go:356] "Generic (PLEG): container finished" podID="9ac9fb68-1982-42a6-8a34-7b9fa61a7565" containerID="5e86837f09aaee2771e702ae294b6171a8cbc15fa300494b81f23b5d1390003b" exitCode=0 Oct 30 23:59:21.172599 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.172579 5577 generic.go:356] "Generic (PLEG): container finished" podID="3d17d96d9d9d049ba1dc638bf8eda6fd" containerID="20f9390a59ceb1c61fe2e7724434ea46acfda811348998c823e582a8b7d4c5f2" exitCode=0 Oct 30 23:59:21.198951 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.198928 5577 manager.go:324] Recovery completed Oct 30 23:59:21.221647 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221587 5577 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:59:21.221647 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221603 5577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:59:21.221647 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221619 5577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221761 5577 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221772 5577 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221788 5577 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221794 5577 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221799 5577 policy_none.go:50] "Start" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221808 5577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:59:21.221938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.221817 5577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:59:21.222995 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.222982 5577 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:59:21.223028 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.222997 5577 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:59:21.223028 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.223003 5577 policy_none.go:44] "Start" Oct 30 23:59:21.224201 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.224185 5577 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:59:21.227429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.227403 5577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:59:21.230489 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.230475 5577 manager.go:341] "Starting Device Plugin manager" Oct 30 23:59:21.231181 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.230500 5577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:59:21.231181 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231182 5577 server.go:87] "Starting device plugin registration server" Oct 30 23:59:21.231486 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231470 5577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:59:21.231551 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231484 5577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:59:21.231602 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231561 5577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:59:21.231691 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231655 5577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:59:21.231768 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231744 5577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:59:21.231768 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231753 5577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:59:21.231896 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.231816 5577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:59:21.231688986Z","UUID":"5725ed7b-2640-418c-bb3e-9e127b959f7d","Handler":null,"Name":"","Endpoint":""} Oct 30 23:59:21.233243 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.233226 5577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:59:21.233496 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.233480 5577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:59:21.233553 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.233506 5577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:59:21.332280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.332253 5577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:59:21.336140 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.336127 5577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:59:21.336204 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.336153 5577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:59:21.336204 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.336168 5577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-236.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:59:21.336284 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.336213 5577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.348353 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.348338 5577 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.348426 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.348392 5577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.425194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425150 5577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal"] Oct 30 23:59:21.425837 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425819 5577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.425932 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425920 5577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.425990 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425920 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"bba88490f41dac4679774c025c13b2ce","Type":"ContainerStarted","Data":"fe20d33a20f01d6bfbee4acfb501f25d7dd765c74bcbda996527c4401a52605a"} Oct 30 23:59:21.425990 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425974 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"bba88490f41dac4679774c025c13b2ce","Type":"ContainerStarted","Data":"41f477e0f70031a637b3539772e17fc709be7738e72639e48f29e0edc4194a7f"} Oct 30 23:59:21.426074 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.425993 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerStarted","Data":"33cb4d68d59200951e35b137c17895fa81224144ba2cd6a86c2dca03224510f4"} Oct 30 23:59:21.426074 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.426013 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerDied","Data":"20f9390a59ceb1c61fe2e7724434ea46acfda811348998c823e582a8b7d4c5f2"} Oct 30 23:59:21.426074 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.426032 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" event={"ID":"3d17d96d9d9d049ba1dc638bf8eda6fd","Type":"ContainerStarted","Data":"d2633f953dc890f18fd2d09d65c8091543349707b5e18031132bb02af2a58052"} Oct 30 23:59:21.441022 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.441003 5577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:21.441073 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.441036 5577 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.441879 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.441867 5577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:21.441926 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:21.441892 5577 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.501140 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.501123 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.501205 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.501151 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.501205 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.501170 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601463 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601437 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601463 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601463 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601600 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601479 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601600 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601518 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bba88490f41dac4679774c025c13b2ce-config\") pod \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"bba88490f41dac4679774c025c13b2ce\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601600 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601552 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:21.601600 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:21.601556 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3d17d96d9d9d049ba1dc638bf8eda6fd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" (UID: \"3d17d96d9d9d049ba1dc638bf8eda6fd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:22.072188 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.072154 5577 apiserver.go:52] "Watching apiserver" Oct 30 23:59:22.081344 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.081320 5577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:59:22.082023 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.081998 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9","openshift-multus/multus-lrwwf","openshift-dns/dns-default-gg8bt","openshift-multus/multus-additional-cni-plugins-cjdd8","kube-system/global-pull-secret-syncer-zqb2v","openshift-multus/network-metrics-daemon-8brkm","kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal","openshift-insights/insights-runtime-extractor-d8zkg","openshift-image-registry/node-ca-7k929","openshift-ovn-kubernetes/ovnkube-node-t2vhd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal","openshift-monitoring/thanos-querier-84b5d76994-jq7jv","openshift-network-diagnostics/network-check-target-2wlm9","openshift-dns/node-resolver-mmztz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4","openshift-ingress-canary/ingress-canary-8vlhd","openshift-monitoring/node-exporter-mddnv","openshift-network-operator/iptables-alerter-xf9b7","openshift-monitoring/prometheus-k8s-0","kube-system/konnectivity-agent-hbqzw","openshift-cluster-node-tuning-operator/tuned-t77d6","openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:22.087195 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.087160 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.087280 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.087251 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.087481 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.087467 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-fl8mx\"" Oct 30 23:59:22.089158 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089137 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.089265 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089233 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.089265 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089243 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:59:22.089441 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089420 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:59:22.089491 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089426 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9ccbg\"" Oct 30 23:59:22.089532 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089488 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:59:22.089579 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089563 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.089614 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089575 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:59:22.089614 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089587 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-rqcpl\"" Oct 30 23:59:22.089614 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089609 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-mmfft\"" Oct 30 23:59:22.089811 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089794 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-g645g\"" Oct 30 23:59:22.089869 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089796 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.089925 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.089889 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:59:22.090032 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090018 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xfgqn\"" Oct 30 23:59:22.090089 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090045 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:59:22.090176 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090162 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:59:22.090229 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090217 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-6ff8s\"" Oct 30 23:59:22.090268 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090262 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:59:22.090995 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.090980 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.091072 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091053 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-72z8g\"" Oct 30 23:59:22.091161 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091108 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:59:22.091213 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091199 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.091770 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091754 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.091770 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091764 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r8kbq\"" Oct 30 23:59:22.091870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091764 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:59:22.091870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.091805 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.092205 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092189 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:59:22.092462 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092448 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:59:22.092522 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092487 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.092565 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092541 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-dtg7z\"" Oct 30 23:59:22.092755 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092738 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:59:22.092836 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.092742 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.093057 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093039 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:59:22.093131 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093061 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.093131 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093068 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:59:22.093131 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093080 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-hrv6t\"" Oct 30 23:59:22.093131 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093103 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.093497 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093482 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pxpmk\"" Oct 30 23:59:22.093565 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093525 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.093621 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.093596 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.094160 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094145 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:59:22.094311 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094289 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-vmg6c\"" Oct 30 23:59:22.094379 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094365 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.094428 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094392 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.094818 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094797 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:59:22.094896 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094825 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:59:22.094896 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.094874 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.095202 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.095189 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-l7b2l\"" Oct 30 23:59:22.095318 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.095299 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.095693 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.095676 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:59:22.095765 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.095744 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:59:22.095966 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.095949 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lxfk8\"" Oct 30 23:59:22.096054 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.096037 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:59:22.096138 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.096125 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:59:22.097421 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097053 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:59:22.097421 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097169 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:59:22.097421 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097230 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:59:22.097421 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097260 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-f8xwm\"" Oct 30 23:59:22.097626 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097423 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:59:22.097626 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097535 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:59:22.097626 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097607 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-xcdsb\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.097844 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099203 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099302 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099326 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-94v5k\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099488 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099514 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.099579 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100114 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100228 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100234 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100341 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100510 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100536 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 23:59:22.100605 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100612 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:59:22.101200 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100641 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5mnbc9lu3rrpo\"" Oct 30 23:59:22.101200 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100921 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-nqp6h\"" Oct 30 23:59:22.101200 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.100998 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 23:59:22.103340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103323 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:22.103395 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103347 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbtk9\" (UniqueName: \"kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:22.103395 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103368 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.103395 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103382 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.103524 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103399 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.103524 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103418 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pggm\" (UniqueName: \"kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.103658 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103634 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.103732 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103671 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.103732 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103678 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjlgk\" (UniqueName: \"kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.103800 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103758 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.103800 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103787 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:59:22.103858 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103813 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:59:22.103858 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103838 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:22.103919 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103865 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.103919 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103888 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104013 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103922 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104013 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103935 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.104013 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.103975 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104013 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104011 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104031 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104058 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/118f5c91-d3a9-434b-b931-89be645e00e2-agent-certs\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104071 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104076 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/118f5c91-d3a9-434b-b931-89be645e00e2-konnectivity-ca\") pod \"konnectivity-agent-hbqzw\" (UID: \"118f5c91-d3a9-434b-b931-89be645e00e2\") " pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104093 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104110 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104139 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104160 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.104194 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104186 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8f07f34-fa12-448b-8054-910fcac344a5-metrics-certs\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104211 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-tmp\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104218 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cni-binary-copy\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104268 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104291 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/36bdf75e-c844-4a33-b5dd-ac33878a0186-cni-binary-copy\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104320 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104387 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104431 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104449 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b4382ac5-fea6-4cf2-b1a6-c06514792224-iptables-alerter-script\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104469 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104493 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104509 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104543 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104564 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdxzw\" (UniqueName: \"kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104599 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104616 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104661 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104677 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104787 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104806 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104835 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.104882 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104851 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlx47\" (UniqueName: \"kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104949 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.104980 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105004 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105026 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105049 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105077 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105103 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmw27\" (UniqueName: \"kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105130 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105136 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-tuned\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105153 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105184 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.105228 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105208 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.106215 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105342 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-daemon-config\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.106215 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105523 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-original-pull-secret\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.106215 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.105563 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2k6kb\"" Oct 30 23:59:22.106215 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.106100 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-serviceca\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.114176 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.114158 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjlgk\" (UniqueName: \"kubernetes.io/projected/2026287d-f8ba-4ba5-9681-d6d74a7d1058-kube-api-access-vjlgk\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.114271 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.114233 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pggm\" (UniqueName: \"kubernetes.io/projected/36bdf75e-c844-4a33-b5dd-ac33878a0186-kube-api-access-6pggm\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.114333 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.114294 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbtk9\" (UniqueName: \"kubernetes.io/projected/a8f07f34-fa12-448b-8054-910fcac344a5-kube-api-access-tbtk9\") pod \"network-metrics-daemon-8brkm\" (UID: \"a8f07f34-fa12-448b-8054-910fcac344a5\") " pod="openshift-multus/network-metrics-daemon-8brkm" Oct 30 23:59:22.114792 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.114773 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4598\" (UniqueName: \"kubernetes.io/projected/02b16bc6-d881-4af8-b248-72a6455b35cd-kube-api-access-m4598\") pod \"network-check-target-2wlm9\" (UID: \"02b16bc6-d881-4af8-b248-72a6455b35cd\") " pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:22.115265 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.115245 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdxzw\" (UniqueName: \"kubernetes.io/projected/b4382ac5-fea6-4cf2-b1a6-c06514792224-kube-api-access-hdxzw\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.116182 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.116163 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlx47\" (UniqueName: \"kubernetes.io/projected/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-kube-api-access-mlx47\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.116244 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.116197 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmw27\" (UniqueName: \"kubernetes.io/projected/ec194b59-61ad-4c7f-b93c-1417107e0269-kube-api-access-wmw27\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.186373 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.186348 5577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:22.186496 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.186482 5577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:22.188679 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188655 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:59:22.188782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188754 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:59:22.188872 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188851 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:59:22.188933 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188877 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:59:22.188984 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188925 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:59:22.189015 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.188996 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:59:22.189076 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189016 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:59:22.189076 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189023 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:59:22.189180 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189110 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:59:22.189336 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189301 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:59:22.189336 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189336 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:59:22.189336 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189353 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fqabajevadsu4\"" Oct 30 23:59:22.189606 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189490 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:59:22.189606 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.189501 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:59:22.190136 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.190122 5577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:59:22.193093 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.193068 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:59:22.193150 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.193117 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:59:22.200654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.200631 5577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:22.200741 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:22.200669 5577 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:22.204442 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.204423 5577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:59:22.204578 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:22.204466 5577 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-236.us-east-2.compute.internal" Oct 30 23:59:22.205679 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205658 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.205788 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205684 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.205788 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205729 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.205788 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205747 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xb7l\" (UniqueName: \"kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.205788 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205762 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.205996 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205801 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.205996 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205828 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-hostroot\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.205996 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205837 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.205996 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205908 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.205996 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.205939 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206207 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206019 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.206207 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206029 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.206207 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206070 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-config\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.206207 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206110 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-run\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206219 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-accelerators-collector-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206231 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206266 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovnkube-script-lib\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206272 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206288 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-var-lib-kubelet\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206327 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206347 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.206407 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206380 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206426 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206504 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-kubelet-config\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206523 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-metrics-client-ca\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206543 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206599 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206632 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206656 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206665 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206688 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-modprobe-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206680 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206734 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-grpc-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206750 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206791 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.206821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206828 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206978 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.206994 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvjn\" (UniqueName: \"kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207056 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/692f415f-94bd-4fd2-ba6c-468145504bc0-metrics-tls\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207072 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207167 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207195 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-multus-certs\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.207429 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207414 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.207658 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207420 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.207695 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207229 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.207695 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207691 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.207777 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207730 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.207777 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207754 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.207848 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207775 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.207848 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207794 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.207848 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207816 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.207848 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207838 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.207988 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207857 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.207988 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207877 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.207988 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207905 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.207988 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207926 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.207988 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207950 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65bn7\" (UniqueName: \"kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.207986 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208016 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208038 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208053 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208074 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:59:22.208135 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208103 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.208307 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208245 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.208307 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208292 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.208369 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208359 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.208409 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208385 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5c12cca1-08f4-49f6-a54d-a02e676f6b54-cert\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:59:22.208447 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208420 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccc77441-8a6d-4a2a-95ca-553f8870b21c-tmp-dir\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.208752 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208574 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.208973 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-metrics-client-ca\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209033 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209104 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209161 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209256 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209342 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209361 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209452 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b4382ac5-fea6-4cf2-b1a6-c06514792224-host-slash\") pod \"iptables-alerter-xf9b7\" (UID: \"b4382ac5-fea6-4cf2-b1a6-c06514792224\") " pod="openshift-network-operator/iptables-alerter-xf9b7" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209487 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-socket-dir-parent\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209661 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.209913 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209748 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.209945 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210004 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210067 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210180 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210219 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210221 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210280 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-etc-kubernetes\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210281 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210333 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210364 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210393 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.210437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210420 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210445 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210477 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210500 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-d\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210540 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-kubernetes\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210592 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210782 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.210964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.210831 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-netns\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.211272 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211101 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-tls\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211522 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-tmp\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211543 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211617 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211665 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211756 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211776 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211779 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211823 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211837 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-sys\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211851 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211914 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.211918 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212058 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212072 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/692f415f-94bd-4fd2-ba6c-468145504bc0-config-volume\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212108 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/99dfb869-c9bf-41d2-832a-d69ae57a802b-env-overrides\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212156 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212230 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212266 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212277 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212348 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212356 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-textfile\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.212433 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212390 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212534 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212570 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212624 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-bin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212667 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndwrv\" (UniqueName: \"kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.212922 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213233 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213293 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213329 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts78k\" (UniqueName: \"kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213375 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.213402 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213398 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213430 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213485 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213536 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213569 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213610 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213642 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213686 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2wt\" (UniqueName: \"kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213750 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213777 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.213835 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213830 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213868 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213921 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.213974 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214024 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214078 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d62jc\" (UniqueName: \"kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214114 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214169 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214205 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:22.214264 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214237 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214296 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214330 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214372 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214403 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214441 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214474 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214523 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214552 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214575 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214606 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.214654 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214636 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214667 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214696 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmj5\" (UniqueName: \"kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214740 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214773 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214806 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214842 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214872 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.214932 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.215139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215041 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5b456bf9-fa54-4740-998e-d0788f096c3c-data-volume\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.215560 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215141 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-systemd\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.215998 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215382 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-run-k8s-cni-cncf-io\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.215998 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215389 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215987 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5b456bf9-fa54-4740-998e-d0788f096c3c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215430 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-cni-multus\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215513 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/692f415f-94bd-4fd2-ba6c-468145504bc0-tmp-dir\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216018 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-cnibin\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215625 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215788 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215832 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysctl-conf\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215453 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-lib-modules\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215950 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215965 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/85bd67fa-8a46-4098-a7b2-0dab23bf0da1-host\") pod \"node-ca-7k929\" (UID: \"85bd67fa-8a46-4098-a7b2-0dab23bf0da1\") " pod="openshift-image-registry/node-ca-7k929" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215973 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/99dfb869-c9bf-41d2-832a-d69ae57a802b-ovn-node-metrics-cert\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215986 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/dfd6c668-59d3-4840-8ccc-e4e9c7066b44-dbus\") pod \"global-pull-secret-syncer-zqb2v\" (UID: \"dfd6c668-59d3-4840-8ccc-e4e9c7066b44\") " pod="kube-system/global-pull-secret-syncer-zqb2v" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215614 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-cnibin\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.216139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216141 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.215859 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216132 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216072 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-multus-conf-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216159 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216176 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-host\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216230 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216246 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216284 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216315 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216338 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fv6v\" (UniqueName: \"kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216348 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/fe424bc5-4967-481f-bc33-3f7b3db28a9e-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216354 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216311 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/36bdf75e-c844-4a33-b5dd-ac33878a0186-system-cni-dir\") pod \"multus-additional-cni-plugins-cjdd8\" (UID: \"36bdf75e-c844-4a33-b5dd-ac33878a0186\") " pod="openshift-multus/multus-additional-cni-plugins-cjdd8" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216383 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216407 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216440 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-host-var-lib-kubelet\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216498 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216526 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216649 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-tls\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216664 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216671 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ec194b59-61ad-4c7f-b93c-1417107e0269-etc-sysconfig\") pod \"tuned-t77d6\" (UID: \"ec194b59-61ad-4c7f-b93c-1417107e0269\") " pod="openshift-cluster-node-tuning-operator/tuned-t77d6" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216692 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216694 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2026287d-f8ba-4ba5-9681-d6d74a7d1058-system-cni-dir\") pod \"multus-lrwwf\" (UID: \"2026287d-f8ba-4ba5-9681-d6d74a7d1058\") " pod="openshift-multus/multus-lrwwf" Oct 30 23:59:22.216870 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.216815 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.219596 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.219573 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.223130 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.223112 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xb7l\" (UniqueName: \"kubernetes.io/projected/692f415f-94bd-4fd2-ba6c-468145504bc0-kube-api-access-6xb7l\") pod \"dns-default-gg8bt\" (UID: \"692f415f-94bd-4fd2-ba6c-468145504bc0\") " pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:22.225375 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.225358 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65bn7\" (UniqueName: \"kubernetes.io/projected/5b456bf9-fa54-4740-998e-d0788f096c3c-kube-api-access-65bn7\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.225739 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.225696 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"prometheus-k8s-0\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:22.226014 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.225993 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"alertmanager-main-0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:22.226129 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.226112 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvjn\" (UniqueName: \"kubernetes.io/projected/5c12cca1-08f4-49f6-a54d-a02e676f6b54-kube-api-access-npvjn\") pod \"ingress-canary-8vlhd\" (UID: \"5c12cca1-08f4-49f6-a54d-a02e676f6b54\") " pod="openshift-ingress-canary/ingress-canary-8vlhd" Oct 30 23:59:22.226272 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.226256 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2wt\" (UniqueName: \"kubernetes.io/projected/ccc77441-8a6d-4a2a-95ca-553f8870b21c-kube-api-access-mq2wt\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.227066 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.227048 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fv6v\" (UniqueName: \"kubernetes.io/projected/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-kube-api-access-6fv6v\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.227291 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.227269 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d62jc\" (UniqueName: \"kubernetes.io/projected/fe424bc5-4967-481f-bc33-3f7b3db28a9e-kube-api-access-d62jc\") pod \"monitoring-plugin-77f4f76594-rjxm9\" (UID: \"fe424bc5-4967-481f-bc33-3f7b3db28a9e\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:22.227343 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.227322 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts78k\" (UniqueName: \"kubernetes.io/projected/c869979e-11d8-4619-8c86-04d2b4acedd6-kube-api-access-ts78k\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.227380 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.227344 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndwrv\" (UniqueName: \"kubernetes.io/projected/6aeaa875-fa12-46cc-b25d-453ce96d4a1e-kube-api-access-ndwrv\") pod \"thanos-querier-84b5d76994-jq7jv\" (UID: \"6aeaa875-fa12-46cc-b25d-453ce96d4a1e\") " pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:22.227419 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.227405 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmj5\" (UniqueName: \"kubernetes.io/projected/99dfb869-c9bf-41d2-832a-d69ae57a802b-kube-api-access-hcmj5\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317580 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317552 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.317580 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317580 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317595 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317609 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317627 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317646 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317656 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-etc-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317677 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317684 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ccc77441-8a6d-4a2a-95ca-553f8870b21c-hosts-file\") pod \"node-resolver-mmztz\" (UID: \"ccc77441-8a6d-4a2a-95ca-553f8870b21c\") " pod="openshift-dns/node-resolver-mmztz" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317701 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-sys-fs\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317704 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-netd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317733 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317742 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-etc-selinux\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317752 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-device-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317776 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.317804 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317787 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317813 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317833 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317838 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-kubelet\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317780 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-ovn\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317859 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317884 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-node-exporter-wtmp\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317885 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317910 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317885 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317920 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-root\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317916 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-log-socket\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317932 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317947 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9ac9fb68-1982-42a6-8a34-7b9fa61a7565-sys\") pod \"node-exporter-mddnv\" (UID: \"9ac9fb68-1982-42a6-8a34-7b9fa61a7565\") " pod="openshift-monitoring/node-exporter-mddnv" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317953 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317961 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-systemd-units\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317967 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.317978 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-registration-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318001 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318005 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5b456bf9-fa54-4740-998e-d0788f096c3c-crio-socket\") pod \"insights-runtime-extractor-d8zkg\" (UID: \"5b456bf9-fa54-4740-998e-d0788f096c3c\") " pod="openshift-insights/insights-runtime-extractor-d8zkg" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318019 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318032 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-slash\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318035 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318054 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-systemd\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318063 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318086 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318092 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318113 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318130 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318149 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318094 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-run-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318169 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318999 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318183 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-cni-bin\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318999 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318196 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c869979e-11d8-4619-8c86-04d2b4acedd6-socket-dir\") pod \"aws-ebs-csi-driver-node-8hhm4\" (UID: \"c869979e-11d8-4619-8c86-04d2b4acedd6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8hhm4" Oct 30 23:59:22.318999 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318201 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-host-run-netns\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318999 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318229 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-node-log\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.318999 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.318257 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/99dfb869-c9bf-41d2-832a-d69ae57a802b-var-lib-openvswitch\") pod \"ovnkube-node-t2vhd\" (UID: \"99dfb869-c9bf-41d2-832a-d69ae57a802b\") " pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:22.483854 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:22.483788 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-rjxm9" Oct 30 23:59:23.392793 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:23.392763 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:23.412180 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:23.412154 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:24.491916 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:24.491874 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:24.572208 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:24.572179 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:24.748461 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:24.748382 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:25.287826 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:25.287793 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gg8bt" Oct 30 23:59:25.450813 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:25.450788 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-84b5d76994-jq7jv" Oct 30 23:59:29.292127 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:29.292041 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 30 23:59:30.266669 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:30.266641 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:31.116749 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:31.116690 5577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:31.293272 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:31.293241 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:31.293554 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:31.293538 5577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:59:32.626066 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:32.626037 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-t2vhd" Oct 30 23:59:32.630726 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:32.630679 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:32.757197 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:32.757160 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:40.500338 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:40.500302 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:40.628938 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:40.628905 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:42.339500 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:42.339467 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:42.465831 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:42.465798 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:42.697921 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:42.697830 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:42.995139 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:42.995066 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-hbqzw" Oct 30 23:59:43.593750 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:43.593699 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:43.721858 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:43.721823 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:44.746362 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746329 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:44.746866 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746809 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="alertmanager" containerID="cri-o://343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" gracePeriod=120 Oct 30 23:59:44.746866 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746846 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy" containerID="cri-o://740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" gracePeriod=120 Oct 30 23:59:44.746866 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746864 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-web" containerID="cri-o://612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" gracePeriod=120 Oct 30 23:59:44.747197 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746893 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="config-reloader" containerID="cri-o://8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" gracePeriod=120 Oct 30 23:59:44.747197 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.746845 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="prom-label-proxy" containerID="cri-o://f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" gracePeriod=120 Oct 30 23:59:44.747197 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:44.747011 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-metric" containerID="cri-o://d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" gracePeriod=120 Oct 30 23:59:45.277782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277750 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" exitCode=0 Oct 30 23:59:45.277782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277778 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" exitCode=0 Oct 30 23:59:45.277782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277784 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" exitCode=0 Oct 30 23:59:45.277782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277792 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" exitCode=0 Oct 30 23:59:45.278039 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277820 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5"} Oct 30 23:59:45.278039 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277886 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed"} Oct 30 23:59:45.278039 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277916 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2"} Oct 30 23:59:45.278039 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:45.277939 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154"} Oct 30 23:59:46.026113 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.026085 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.195455 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195381 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195455 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195413 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195455 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195445 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195466 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195501 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195528 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195552 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195583 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195623 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195655 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.195730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195693 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.196158 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195797 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.196158 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.195834 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") pod \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\" (UID: \"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0\") " Oct 30 23:59:46.196158 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.196020 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:46.196158 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.196102 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-main-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.197560 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.197526 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:46.198647 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.198621 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:46.198926 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.198877 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.199049 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.198975 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.199049 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.199011 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:46.199151 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.199083 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.199439 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.199329 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "kube-api-access-767n5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:46.199439 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.199382 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.200118 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.200092 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.201032 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.201001 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:46.204222 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.204199 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.209875 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.209855 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config" pod "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" (UID: "0d65c50f-29cb-4be0-8759-8a19c4f1b7d0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:46.220510 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.220484 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:46.284838 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.284813 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" exitCode=0 Oct 30 23:59:46.284838 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.284836 5577 generic.go:356] "Generic (PLEG): container finished" podID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerID="612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" exitCode=0 Oct 30 23:59:46.284979 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.284896 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:59:46.284979 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.284889 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb"} Oct 30 23:59:46.285040 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.285028 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e"} Oct 30 23:59:46.285076 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.285054 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0d65c50f-29cb-4be0-8759-8a19c4f1b7d0","Type":"ContainerDied","Data":"4324e86af798960ca6ad1905f431bbedbaa48815ba2f9080d99acdf3b2459e22"} Oct 30 23:59:46.285107 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.285084 5577 scope.go:122] "RemoveContainer" containerID="f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" Oct 30 23:59:46.293230 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.293214 5577 scope.go:122] "RemoveContainer" containerID="d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" Oct 30 23:59:46.296404 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296384 5577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-main-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296406 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296416 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-767n5\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-kube-api-access-767n5\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296425 5577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296434 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296443 5577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296452 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296462 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296471 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296480 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296479 5577 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-metrics-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296487 5577 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.296821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.296495 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:46.311143 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.311118 5577 scope.go:122] "RemoveContainer" containerID="740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" Oct 30 23:59:46.312192 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.312165 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:46.318928 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.318865 5577 scope.go:122] "RemoveContainer" containerID="612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" Oct 30 23:59:46.319616 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.319597 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:59:46.325394 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.325379 5577 scope.go:122] "RemoveContainer" containerID="8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" Oct 30 23:59:46.331940 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.331926 5577 scope.go:122] "RemoveContainer" containerID="343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" Oct 30 23:59:46.338514 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.338498 5577 scope.go:122] "RemoveContainer" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" Oct 30 23:59:46.345218 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.345199 5577 scope.go:122] "RemoveContainer" containerID="f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" Oct 30 23:59:46.345505 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.345487 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5\": container with ID starting with f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5 not found: ID does not exist" containerID="f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" Oct 30 23:59:46.345557 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.345513 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5"} err="failed to get container status \"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5\": rpc error: code = NotFound desc = could not find container \"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5\": container with ID starting with f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5 not found: ID does not exist" Oct 30 23:59:46.345557 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.345543 5577 scope.go:122] "RemoveContainer" containerID="d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" Oct 30 23:59:46.345807 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.345785 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb\": container with ID starting with d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb not found: ID does not exist" containerID="d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" Oct 30 23:59:46.345857 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.345817 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb"} err="failed to get container status \"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb\": rpc error: code = NotFound desc = could not find container \"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb\": container with ID starting with d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb not found: ID does not exist" Oct 30 23:59:46.345857 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.345840 5577 scope.go:122] "RemoveContainer" containerID="740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" Oct 30 23:59:46.346110 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.346093 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed\": container with ID starting with 740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed not found: ID does not exist" containerID="740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" Oct 30 23:59:46.346152 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346115 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed"} err="failed to get container status \"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed\": rpc error: code = NotFound desc = could not find container \"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed\": container with ID starting with 740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed not found: ID does not exist" Oct 30 23:59:46.346152 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346129 5577 scope.go:122] "RemoveContainer" containerID="612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" Oct 30 23:59:46.346343 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.346327 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e\": container with ID starting with 612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e not found: ID does not exist" containerID="612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" Oct 30 23:59:46.346384 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346348 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e"} err="failed to get container status \"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e\": rpc error: code = NotFound desc = could not find container \"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e\": container with ID starting with 612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e not found: ID does not exist" Oct 30 23:59:46.346384 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346360 5577 scope.go:122] "RemoveContainer" containerID="8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" Oct 30 23:59:46.346595 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.346579 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2\": container with ID starting with 8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2 not found: ID does not exist" containerID="8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" Oct 30 23:59:46.346636 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346601 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2"} err="failed to get container status \"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2\": rpc error: code = NotFound desc = could not find container \"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2\": container with ID starting with 8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2 not found: ID does not exist" Oct 30 23:59:46.346636 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346617 5577 scope.go:122] "RemoveContainer" containerID="343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" Oct 30 23:59:46.346852 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.346835 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154\": container with ID starting with 343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154 not found: ID does not exist" containerID="343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" Oct 30 23:59:46.346903 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346856 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154"} err="failed to get container status \"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154\": rpc error: code = NotFound desc = could not find container \"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154\": container with ID starting with 343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154 not found: ID does not exist" Oct 30 23:59:46.346903 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.346868 5577 scope.go:122] "RemoveContainer" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" Oct 30 23:59:46.347097 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:46.347081 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190\": container with ID starting with 34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190 not found: ID does not exist" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" Oct 30 23:59:46.347137 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347102 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190"} err="failed to get container status \"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190\": rpc error: code = NotFound desc = could not find container \"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190\": container with ID starting with 34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190 not found: ID does not exist" Oct 30 23:59:46.347137 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347116 5577 scope.go:122] "RemoveContainer" containerID="f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5" Oct 30 23:59:46.347322 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347305 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5"} err="failed to get container status \"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5\": rpc error: code = NotFound desc = could not find container \"f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5\": container with ID starting with f5a62ffa4319506009316da87d92cb6fbac9e8917eb030961ae2b1a1ffd99bb5 not found: ID does not exist" Oct 30 23:59:46.347398 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347323 5577 scope.go:122] "RemoveContainer" containerID="d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb" Oct 30 23:59:46.347516 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347498 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb"} err="failed to get container status \"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb\": rpc error: code = NotFound desc = could not find container \"d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb\": container with ID starting with d75684d5670eae5a80ca3b2509e94f045205b75f27623850073c34e58299c9bb not found: ID does not exist" Oct 30 23:59:46.347559 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347517 5577 scope.go:122] "RemoveContainer" containerID="740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed" Oct 30 23:59:46.347692 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347676 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed"} err="failed to get container status \"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed\": rpc error: code = NotFound desc = could not find container \"740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed\": container with ID starting with 740eb54680100d8e435c64ffa3cac5d932dc2a03c36a9e453489dda5c78303ed not found: ID does not exist" Oct 30 23:59:46.347802 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347694 5577 scope.go:122] "RemoveContainer" containerID="612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e" Oct 30 23:59:46.347919 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347903 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e"} err="failed to get container status \"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e\": rpc error: code = NotFound desc = could not find container \"612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e\": container with ID starting with 612fd421049d3e043c757af3433811b5fa2f921a8cfee3e531d1e8e9c1aaf86e not found: ID does not exist" Oct 30 23:59:46.347959 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.347919 5577 scope.go:122] "RemoveContainer" containerID="8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2" Oct 30 23:59:46.348095 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.348078 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2"} err="failed to get container status \"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2\": rpc error: code = NotFound desc = could not find container \"8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2\": container with ID starting with 8cd6c86764fb8f060d22f5cc5acbd28d0e8d00beea170ea4388fa515347d3fc2 not found: ID does not exist" Oct 30 23:59:46.348095 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.348094 5577 scope.go:122] "RemoveContainer" containerID="343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154" Oct 30 23:59:46.348328 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.348309 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154"} err="failed to get container status \"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154\": rpc error: code = NotFound desc = could not find container \"343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154\": container with ID starting with 343af250f8263120e8dd064af67ff6d7ded46488452d576738fdc51fce844154 not found: ID does not exist" Oct 30 23:59:46.348328 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.348327 5577 scope.go:122] "RemoveContainer" containerID="34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190" Oct 30 23:59:46.348591 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.348573 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190"} err="failed to get container status \"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190\": rpc error: code = NotFound desc = could not find container \"34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190\": container with ID starting with 34567609d551ffcdfa2d7ac7b4566df9ce1c92935ca45eb357213145d612d190 not found: ID does not exist" Oct 30 23:59:46.349593 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:46.349573 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:47.128626 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:47.128550 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" path="/var/lib/kubelet/pods/0d65c50f-29cb-4be0-8759-8a19c4f1b7d0/volumes" Oct 30 23:59:49.016217 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016182 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:49.016612 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016509 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="prometheus" containerID="cri-o://ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" gracePeriod=600 Oct 30 23:59:49.016612 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016535 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy" containerID="cri-o://dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" gracePeriod=600 Oct 30 23:59:49.016612 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016559 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-thanos" containerID="cri-o://623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" gracePeriod=600 Oct 30 23:59:49.016821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016590 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="config-reloader" containerID="cri-o://cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" gracePeriod=600 Oct 30 23:59:49.016821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016679 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-web" containerID="cri-o://04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" gracePeriod=600 Oct 30 23:59:49.016821 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.016758 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="thanos-sidecar" containerID="cri-o://88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" gracePeriod=600 Oct 30 23:59:49.247662 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.247637 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.302784 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302696 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" exitCode=0 Oct 30 23:59:49.302784 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302741 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" exitCode=0 Oct 30 23:59:49.302784 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302750 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" exitCode=0 Oct 30 23:59:49.302784 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302746 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302803 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302834 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302851 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302877 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302759 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" exitCode=0 Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302930 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302938 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" exitCode=0 Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302955 5577 generic.go:356] "Generic (PLEG): container finished" podID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" exitCode=0 Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.302972 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.303012 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} Oct 30 23:59:49.303071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.303052 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a6948ba0-e600-4cfe-8e6d-e8e912da4861","Type":"ContainerDied","Data":"d2622315834f4b220ccaecbf64a273de7061a2258de62299ea7b3372d9f2de15"} Oct 30 23:59:49.310502 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.310484 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.316868 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.316839 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.323014 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.322999 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.329036 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.329019 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.335311 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.335290 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.341675 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.341656 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.347660 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.347646 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.347965 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.347947 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.348050 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.347977 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.348050 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348008 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.348275 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.348258 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.348319 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348281 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.348319 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348297 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.348576 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.348562 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.348644 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348577 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.348644 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348587 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.348873 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.348856 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.348916 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348880 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.348916 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.348896 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.349127 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.349112 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.349166 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349132 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.349166 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349144 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.349376 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.349361 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.349426 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349379 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.349426 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349390 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.349635 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:49.349620 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.349678 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349640 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.349678 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349655 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.349899 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349883 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.349964 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.349900 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.350132 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350113 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.350132 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350131 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.350344 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350329 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.350344 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350343 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.350549 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350534 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.350549 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350549 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.350782 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350764 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.350823 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.350782 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.351027 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351009 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.351071 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351028 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.351245 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351228 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.351341 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351245 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.351474 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351456 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.351514 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351475 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.351692 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351671 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.351785 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351694 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.351960 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351942 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.352008 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.351961 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.352190 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352176 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.352232 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352191 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.352398 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352379 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.352446 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352401 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.352591 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352571 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.352627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352591 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.352828 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352803 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.352828 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.352821 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.353041 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353025 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.353088 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353041 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.353267 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353252 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.353308 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353267 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.353507 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353490 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.353548 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353508 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.353738 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353721 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.353791 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353738 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.353941 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353919 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.353980 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.353941 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.354167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354150 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.354215 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354167 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.354401 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354379 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.354401 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354393 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.354612 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354592 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.354612 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354611 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.354879 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354860 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.354923 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.354881 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.355112 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355093 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.355173 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355114 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.355347 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355331 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.355390 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355348 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.355548 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355532 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.355593 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355549 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.355795 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355774 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.355876 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.355797 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.356081 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356055 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.356120 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356081 5577 scope.go:122] "RemoveContainer" containerID="623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3" Oct 30 23:59:49.356301 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356280 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3"} err="failed to get container status \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": rpc error: code = NotFound desc = could not find container \"623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3\": container with ID starting with 623194e417c1558f0c765b0aa901f93a46323ed9c038a3a98a941cf480bb45f3 not found: ID does not exist" Oct 30 23:59:49.356343 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356302 5577 scope.go:122] "RemoveContainer" containerID="dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7" Oct 30 23:59:49.356491 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356476 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7"} err="failed to get container status \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": rpc error: code = NotFound desc = could not find container \"dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7\": container with ID starting with dbbe9b0a74eb990394d71ef9806ce0772b86e9e621f73bcd8f09ada75e27a3a7 not found: ID does not exist" Oct 30 23:59:49.356491 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356491 5577 scope.go:122] "RemoveContainer" containerID="04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404" Oct 30 23:59:49.356666 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356647 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404"} err="failed to get container status \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": rpc error: code = NotFound desc = could not find container \"04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404\": container with ID starting with 04533399cd4250732b5d1d531f97b99a8c05dfa622628185a9778da27ea14404 not found: ID does not exist" Oct 30 23:59:49.356730 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356666 5577 scope.go:122] "RemoveContainer" containerID="88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f" Oct 30 23:59:49.356901 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356884 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f"} err="failed to get container status \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": rpc error: code = NotFound desc = could not find container \"88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f\": container with ID starting with 88a67022e41d1c79116bb66369b1acdf2e86fb4107c9fe9edee0dc833167812f not found: ID does not exist" Oct 30 23:59:49.356958 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.356901 5577 scope.go:122] "RemoveContainer" containerID="cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208" Oct 30 23:59:49.357102 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.357087 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208"} err="failed to get container status \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": rpc error: code = NotFound desc = could not find container \"cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208\": container with ID starting with cea77a1f4f81925daeb0317bc17ec9fb7f3d7850d72622e4123f37b12e220208 not found: ID does not exist" Oct 30 23:59:49.357151 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.357102 5577 scope.go:122] "RemoveContainer" containerID="ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155" Oct 30 23:59:49.357297 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.357277 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155"} err="failed to get container status \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": rpc error: code = NotFound desc = could not find container \"ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155\": container with ID starting with ff616d3c6d4ef14701823d939ccbd50df98091c008906d8306efa3f342590155 not found: ID does not exist" Oct 30 23:59:49.357297 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.357291 5577 scope.go:122] "RemoveContainer" containerID="f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6" Oct 30 23:59:49.357473 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.357455 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6"} err="failed to get container status \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": rpc error: code = NotFound desc = could not find container \"f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6\": container with ID starting with f8c2f19743dd19159890f94726fe025dceaca796242d8089e8c169eb81bd70a6 not found: ID does not exist" Oct 30 23:59:49.438361 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438339 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438444 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438378 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438444 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438401 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438444 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438420 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438444 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438438 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438457 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438476 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438492 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438549 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438595 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438589 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438622 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438663 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438699 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438739 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438763 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438792 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438834 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.438867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438864 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438897 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438947 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438967 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") pod \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\" (UID: \"a6948ba0-e600-4cfe-8e6d-e8e912da4861\") " Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.438982 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.439162 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.439252 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.439179 5577 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.439931 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.439785 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.440041 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.440013 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.440104 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.440054 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.440560 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.440309 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.441588 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.441032 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:49.441588 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.441271 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.442074 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.441930 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:49.442742 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.442694 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.443294 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443258 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.443386 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443349 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.443532 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443484 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.443610 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443565 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.443665 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443644 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "kube-api-access-8pgql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:49.443748 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443727 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:49.444005 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.443982 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:49.444535 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.444516 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.444820 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.444805 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.451784 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.451747 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config" pod "a6948ba0-e600-4cfe-8e6d-e8e912da4861" (UID: "a6948ba0-e600-4cfe-8e6d-e8e912da4861"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:49.539695 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539665 5577 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539695 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539690 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539695 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539699 5577 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-metrics-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539726 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539736 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539747 5577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539756 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539765 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539773 5577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539782 5577 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-kube-rbac-proxy\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539790 5577 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-grpc-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539800 5577 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539808 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539817 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a6948ba0-e600-4cfe-8e6d-e8e912da4861-prometheus-k8s-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539825 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539832 5577 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-metrics-client-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539840 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pgql\" (UniqueName: \"kubernetes.io/projected/a6948ba0-e600-4cfe-8e6d-e8e912da4861-kube-api-access-8pgql\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.539873 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.539849 5577 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a6948ba0-e600-4cfe-8e6d-e8e912da4861-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:49.594631 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.594579 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:49.627715 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.627692 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:49.633757 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.633737 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:49.671376 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671280 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:49.671631 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671617 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671634 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671642 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="prometheus" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671646 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="prometheus" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671657 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="thanos-sidecar" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671661 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="thanos-sidecar" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671667 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="alertmanager" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671670 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="alertmanager" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671676 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy" Oct 30 23:59:49.671677 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671680 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671686 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="prom-label-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671690 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="prom-label-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671701 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671724 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671730 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671734 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671749 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="init-config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671753 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="init-config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671760 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="init-config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671764 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="init-config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671771 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-thanos" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671774 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-thanos" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671780 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671784 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671792 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-metric" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671795 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-metric" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671802 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671806 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671842 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="init-config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671850 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-thanos" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671855 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671884 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy-metric" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671890 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671897 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy-web" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671902 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="prometheus" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671907 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="thanos-sidecar" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671912 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="alertmanager" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671917 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="kube-rbac-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671922 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="prom-label-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671928 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="kube-rbac-proxy" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671933 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d65c50f-29cb-4be0-8759-8a19c4f1b7d0" containerName="config-reloader" Oct 30 23:59:49.671957 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.671938 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" containerName="init-config-reloader" Oct 30 23:59:49.675030 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.675015 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.679477 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.679456 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 23:59:49.679558 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.679475 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 23:59:49.679558 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.679483 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 23:59:49.682220 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.682203 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 23:59:49.685516 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.685497 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 23:59:49.685516 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.685496 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 23:59:49.685649 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.685535 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 23:59:49.685649 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.685566 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 23:59:49.689437 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.689422 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 23:59:49.689791 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.689774 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 23:59:49.699878 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.699858 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:49.744869 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.744822 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:49.779981 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.779963 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 23:59:49.779981 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.779963 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 23:59:49.780121 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.780036 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 23:59:49.780238 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.780226 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fqabajevadsu4\"" Oct 30 23:59:49.780301 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.780288 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 23:59:49.780337 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.780328 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-2k6kb\"" Oct 30 23:59:49.780382 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.780368 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 23:59:49.842794 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.842770 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.842862 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.842799 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.842862 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.842817 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.842862 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.842835 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.842994 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.842978 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843084 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843005 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843084 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843023 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843084 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843045 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843084 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843070 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7zcz\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-kube-api-access-x7zcz\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843209 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843107 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843209 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843149 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843209 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843176 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843209 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843192 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843232 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843257 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843289 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843316 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843340 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843331 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843350 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.843512 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.843389 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944742 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944658 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944742 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944689 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944742 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944739 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944765 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944793 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944819 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944871 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944901 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.944949 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944929 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.945156 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.944955 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945309 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945434 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945468 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945506 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945532 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7zcz\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-kube-api-access-x7zcz\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945688 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945860 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945900 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.945930 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.946016 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.946259 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.946430 5577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.946455 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.946814 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.947361 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.947519 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.947966 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.949736 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.948696 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.950659 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.950315 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.951857 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.951678 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.952867 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.952808 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.953566 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.953539 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b93e19ea-595f-458b-8919-e718a87cea5f-config-out\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.953660 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.953547 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.953660 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.953646 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.954014 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.953993 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.954149 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.954128 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-web-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.954207 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.954190 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.954242 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.954227 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.955149 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.955135 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b93e19ea-595f-458b-8919-e718a87cea5f-config\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.955502 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.955485 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b93e19ea-595f-458b-8919-e718a87cea5f-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.960235 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.960215 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7zcz\" (UniqueName: \"kubernetes.io/projected/b93e19ea-595f-458b-8919-e718a87cea5f-kube-api-access-x7zcz\") pod \"prometheus-k8s-0\" (UID: \"b93e19ea-595f-458b-8919-e718a87cea5f\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:49.990588 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:49.990572 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:50.277499 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:50.277466 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 23:59:50.280617 ip-10-0-136-236 kubenswrapper[5577]: W1030 23:59:50.280589 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb93e19ea_595f_458b_8919_e718a87cea5f.slice/crio-488d9b7cc7fad2fa217599693c3d6949382f16a0dc0c24c6c5d05084109d4e74 WatchSource:0}: Error finding container 488d9b7cc7fad2fa217599693c3d6949382f16a0dc0c24c6c5d05084109d4e74: Status 404 returned error can't find the container with id 488d9b7cc7fad2fa217599693c3d6949382f16a0dc0c24c6c5d05084109d4e74 Oct 30 23:59:50.307074 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:50.307048 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"488d9b7cc7fad2fa217599693c3d6949382f16a0dc0c24c6c5d05084109d4e74"} Oct 30 23:59:51.128166 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:51.128126 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6948ba0-e600-4cfe-8e6d-e8e912da4861" path="/var/lib/kubelet/pods/a6948ba0-e600-4cfe-8e6d-e8e912da4861/volumes" Oct 30 23:59:51.312962 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:51.312890 5577 generic.go:356] "Generic (PLEG): container finished" podID="b93e19ea-595f-458b-8919-e718a87cea5f" containerID="b26216feab5745cb94cf2fba25470e68e2d08dde25c9284d869ff431185aefbf" exitCode=0 Oct 30 23:59:51.313393 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:51.312962 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerDied","Data":"b26216feab5745cb94cf2fba25470e68e2d08dde25c9284d869ff431185aefbf"} Oct 30 23:59:52.320142 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320104 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"da882af7368c18b06d75f049a5656592a97605b13319244514aa212fa3db5962"} Oct 30 23:59:52.320627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320212 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"5d38b23c2d19cf5cd1b9722549716f42b1002f191c11cd6be0718bb669d644e2"} Oct 30 23:59:52.320627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320257 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"67599b51c3619e05fbfb686a462ba2e9364609c4c522dbc1a15f942ccfdd223b"} Oct 30 23:59:52.320627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320302 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"9c48fd30dc143ee9260d40856c267f103b16527171baf1038bf2cf804dce2592"} Oct 30 23:59:52.320627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320393 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"765921b0f27f9cae4d84a11f91247912bf61dbc6671d2b2ae4f9f0ced907e3e3"} Oct 30 23:59:52.320627 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.320432 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b93e19ea-595f-458b-8919-e718a87cea5f","Type":"ContainerStarted","Data":"0742b1ff2cbb0445bd96d2a98374a656788affe3dccc046eaab77004b2908d31"} Oct 30 23:59:52.358125 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:52.358052 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.343428024 podStartE2EDuration="3.357998394s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.57037ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:52.356690228 +0000 UTC m=+31.348349824" watchObservedRunningTime="2026-10-30 23:59:52.357998394 +0000 UTC m=+31.349657970" Oct 30 23:59:54.494533 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:54.494505 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2wlm9" Oct 30 23:59:54.990756 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:54.990728 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 23:59:57.067684 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:57.067640 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:57.198167 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:57.198126 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:57.507479 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:57.507449 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:57.639465 ip-10-0-136-236 kubenswrapper[5577]: I1030 23:59:57.639431 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 30 23:59:59.292058 ip-10-0-136-236 kubenswrapper[5577]: E1030 23:59:59.292026 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:00:02.640546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:02.640511 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:02.775171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:02.775129 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:09.490863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:09.490827 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:24.685169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:24.685127 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:29.291252 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:00:29.291219 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:00:31.936285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:31.936251 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:00:32.107939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:32.107911 5577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:00:32.123574 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:32.123549 5577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:00:32.461474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:00:32.461446 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 00:00:59.291509 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:00:59.291430 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:01:00.891837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:00.891793 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:24.651382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:24.651347 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:29.291699 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:01:29.291657 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:01:33.527218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:33.527178 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:33.655860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:33.655822 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:41.338378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:41.338345 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:41.486619 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:41.486588 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:56.686373 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:56.686331 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:56.823276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:01:56.823237 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:01:59.291391 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:01:59.291350 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:02:03.501745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:03.501695 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:04.268313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:04.268279 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:06.195383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:06.195349 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:06.323449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:06.323405 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:06.901598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:06.901560 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:07.033172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:07.033132 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:09.412148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:09.412111 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:09.575518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:09.575479 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:11.480922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:11.480882 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:11.610066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:11.610027 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:11.775591 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:11.775561 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:11.998212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:11.998134 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:18.178321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:18.178258 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:18.310781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:18.310743 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:19.068248 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:19.068210 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:24.676047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:24.676013 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:28.644101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.644012 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:28.746411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.746378 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-observability-system/odh-observability-85cd6b576d-z5f58"] Oct 31 00:02:28.749771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.749752 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.760861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.760837 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-85cd6b576d-z5f58"] Oct 31 00:02:28.761411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.761389 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-metrics-tls\"" Oct 31 00:02:28.761533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.761398 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-webhook-cert\"" Oct 31 00:02:28.761603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.761424 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"kube-root-ca.crt\"" Oct 31 00:02:28.761603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.761453 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"openshift-service-ca.crt\"" Oct 31 00:02:28.761603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.761456 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-dmqnv\"" Oct 31 00:02:28.808882 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.808835 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:28.820411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.820386 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg957\" (UniqueName: \"kubernetes.io/projected/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-kube-api-access-wg957\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.820501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.820415 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.820501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.820451 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-metrics-tls\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.921749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.921655 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg957\" (UniqueName: \"kubernetes.io/projected/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-kube-api-access-wg957\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.921749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.921700 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.921903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.921777 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-metrics-tls\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.921903 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:28.921819 5577 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 31 00:02:28.921903 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:28.921886 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs podName:c2d7942b-59c9-4d34-bf75-4156c16e5bbd nodeName:}" failed. No retries permitted until 2026-10-31 00:02:29.421869583 +0000 UTC m=+188.413529178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs") pod "odh-observability-85cd6b576d-z5f58" (UID: "c2d7942b-59c9-4d34-bf75-4156c16e5bbd") : secret "odh-observability-webhook-cert" not found Oct 31 00:02:28.924339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.924313 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-metrics-tls\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:28.934191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.934165 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:28.937068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:28.937052 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg957\" (UniqueName: \"kubernetes.io/projected/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-kube-api-access-wg957\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:29.291481 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:29.291441 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:02:29.425590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:29.425557 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:29.425789 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:29.425686 5577 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 31 00:02:29.425789 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:29.425760 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs podName:c2d7942b-59c9-4d34-bf75-4156c16e5bbd nodeName:}" failed. No retries permitted until 2026-10-31 00:02:30.425744638 +0000 UTC m=+189.417404213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs") pod "odh-observability-85cd6b576d-z5f58" (UID: "c2d7942b-59c9-4d34-bf75-4156c16e5bbd") : secret "odh-observability-webhook-cert" not found Oct 31 00:02:30.432220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:30.432179 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:30.432752 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:30.432348 5577 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 31 00:02:30.432752 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:30.432447 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs podName:c2d7942b-59c9-4d34-bf75-4156c16e5bbd nodeName:}" failed. No retries permitted until 2026-10-31 00:02:32.43242227 +0000 UTC m=+191.424081848 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs") pod "odh-observability-85cd6b576d-z5f58" (UID: "c2d7942b-59c9-4d34-bf75-4156c16e5bbd") : secret "odh-observability-webhook-cert" not found Oct 31 00:02:32.449616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:32.449591 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:32.450016 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:32.449742 5577 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 31 00:02:32.450016 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:32.449796 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs podName:c2d7942b-59c9-4d34-bf75-4156c16e5bbd nodeName:}" failed. No retries permitted until 2026-10-31 00:02:36.449783227 +0000 UTC m=+195.441442789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs") pod "odh-observability-85cd6b576d-z5f58" (UID: "c2d7942b-59c9-4d34-bf75-4156c16e5bbd") : secret "odh-observability-webhook-cert" not found Oct 31 00:02:36.483795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:36.483762 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:36.484310 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:36.483929 5577 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 31 00:02:36.484310 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:36.484026 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs podName:c2d7942b-59c9-4d34-bf75-4156c16e5bbd nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.484004626 +0000 UTC m=+203.475664204 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs") pod "odh-observability-85cd6b576d-z5f58" (UID: "c2d7942b-59c9-4d34-bf75-4156c16e5bbd") : secret "odh-observability-webhook-cert" not found Oct 31 00:02:43.512264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:43.512225 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:02:44.547059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:44.547012 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:44.549637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:44.549612 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/c2d7942b-59c9-4d34-bf75-4156c16e5bbd-webhook-certs\") pod \"odh-observability-85cd6b576d-z5f58\" (UID: \"c2d7942b-59c9-4d34-bf75-4156c16e5bbd\") " pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:44.661911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:44.661881 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:44.935178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:44.935107 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-85cd6b576d-z5f58"] Oct 31 00:02:44.938531 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:02:44.938502 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d7942b_59c9_4d34_bf75_4156c16e5bbd.slice/crio-44b32fe6ae3582135cf70d1421b784eaa0767123f6c0f9d287ddeca9d80877f8 WatchSource:0}: Error finding container 44b32fe6ae3582135cf70d1421b784eaa0767123f6c0f9d287ddeca9d80877f8: Status 404 returned error can't find the container with id 44b32fe6ae3582135cf70d1421b784eaa0767123f6c0f9d287ddeca9d80877f8 Oct 31 00:02:44.940061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:44.940039 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:02:45.867220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:45.867176 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerStarted","Data":"44b32fe6ae3582135cf70d1421b784eaa0767123f6c0f9d287ddeca9d80877f8"} Oct 31 00:02:48.880183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:48.880141 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerStarted","Data":"fefbab087abcbb36cce9c5ca2a51e8677c1552fbc5bd7daa1b06c50e74d894b5"} Oct 31 00:02:48.880569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:48.880289 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:02:48.902083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:48.902040 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" podStartSLOduration=17.987240391 podStartE2EDuration="20.90203044s" totalImagesPullingTime="2.914790049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:48.901915404 +0000 UTC m=+207.893574986" watchObservedRunningTime="2026-10-31 00:02:48.90203044 +0000 UTC m=+207.893690021" Oct 31 00:02:59.291230 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:02:59.291180 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:02:59.885367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:02:59.885336 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:03:05.000983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:03:05.000951 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:05.140146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:03:05.140110 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:07.898549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:03:07.898514 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:08.036019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:03:08.035981 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:24.742820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:03:24.742775 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:03:29.291435 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:03:29.291400 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:03:59.290988 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:03:59.290910 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:04:07.400012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:07.399966 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:08.371468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:08.371426 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:15.103311 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:15.103275 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:15.236078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:15.236031 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:15.598862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:15.598831 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:15.800681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:15.800646 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:16.634578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:16.634549 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:17.507276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:17.507215 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:18.129676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:18.129640 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:18.268195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:18.268160 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:18.412425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:18.412334 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:19.412800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:19.412754 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:19.549961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:19.549919 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:21.150344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:21.150302 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:04:21.150754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:21.150468 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:04:21.233960 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:21.233926 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:21.399950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:21.399919 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:21.543993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:21.543958 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:24.494172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:24.494142 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:24.806321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:24.806288 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:25.196788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:25.196678 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:25.634830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:25.634795 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:25.763032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:25.762973 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:25.990480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:25.990392 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:27.045998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:27.045951 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:27.179111 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:27.179077 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:27.307157 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:27.307073 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:27.853331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:27.853296 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:28.036404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:28.036367 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:28.638384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:28.638329 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:28.780788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:28.780758 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:28.947248 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:28.947173 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:29.291899 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:04:29.291868 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:04:31.409967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.409936 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.574247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.574206 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.712237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.712157 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj"] Oct 31 00:04:31.714837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.714816 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" Oct 31 00:04:31.721740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.721696 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-dockercfg-h8fqx\"" Oct 31 00:04:31.722532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.722515 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:31.722659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.722641 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:31.731473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.731451 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj"] Oct 31 00:04:31.732511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.732484 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.832105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.832074 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx8z7\" (UniqueName: \"kubernetes.io/projected/79046fb0-7fc5-4930-8416-7d9278c90ba9-kube-api-access-gx8z7\") pod \"obo-prometheus-operator-6ddb664c4f-bgdwj\" (UID: \"79046fb0-7fc5-4930-8416-7d9278c90ba9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" Oct 31 00:04:31.866116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.866086 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:31.932388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.932360 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx8z7\" (UniqueName: \"kubernetes.io/projected/79046fb0-7fc5-4930-8416-7d9278c90ba9-kube-api-access-gx8z7\") pod \"obo-prometheus-operator-6ddb664c4f-bgdwj\" (UID: \"79046fb0-7fc5-4930-8416-7d9278c90ba9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" Oct 31 00:04:31.947773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:31.947748 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx8z7\" (UniqueName: \"kubernetes.io/projected/79046fb0-7fc5-4930-8416-7d9278c90ba9-kube-api-access-gx8z7\") pod \"obo-prometheus-operator-6ddb664c4f-bgdwj\" (UID: \"79046fb0-7fc5-4930-8416-7d9278c90ba9\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" Oct 31 00:04:32.026717 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.026687 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" Oct 31 00:04:32.254545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.254517 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7"] Oct 31 00:04:32.257846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.257818 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.273577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.273511 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7"] Oct 31 00:04:32.293908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.293843 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:32.304474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.304453 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj"] Oct 31 00:04:32.305887 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:04:32.305861 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79046fb0_7fc5_4930_8416_7d9278c90ba9.slice/crio-635f4487aa9663ccfe4b70cf74eaea7b99ce3ed2cfcef9af445c87779e0ff9df WatchSource:0}: Error finding container 635f4487aa9663ccfe4b70cf74eaea7b99ce3ed2cfcef9af445c87779e0ff9df: Status 404 returned error can't find the container with id 635f4487aa9663ccfe4b70cf74eaea7b99ce3ed2cfcef9af445c87779e0ff9df Oct 31 00:04:32.336353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.336328 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.336445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.336360 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.354175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.354142 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 31 00:04:32.354316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.354303 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-fgczj\"" Oct 31 00:04:32.437025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.436996 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.437387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.437033 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.439580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.439553 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.439728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.439558 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cc560d9-fc04-4e45-943a-cd074028e2fb-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7\" (UID: \"2cc560d9-fc04-4e45-943a-cd074028e2fb\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.502177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.502147 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:32.569601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.569526 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" Oct 31 00:04:32.851907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:32.851812 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7"] Oct 31 00:04:32.853947 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:04:32.853923 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cc560d9_fc04_4e45_943a_cd074028e2fb.slice/crio-d4c9e1b8571d514cf22adedb11bc5c6c44cd16b2a494bf002d43542ed9cc2855 WatchSource:0}: Error finding container d4c9e1b8571d514cf22adedb11bc5c6c44cd16b2a494bf002d43542ed9cc2855: Status 404 returned error can't find the container with id d4c9e1b8571d514cf22adedb11bc5c6c44cd16b2a494bf002d43542ed9cc2855 Oct 31 00:04:33.133896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.133800 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:33.216177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.216139 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" event={"ID":"2cc560d9-fc04-4e45-943a-cd074028e2fb","Type":"ContainerStarted","Data":"d4c9e1b8571d514cf22adedb11bc5c6c44cd16b2a494bf002d43542ed9cc2855"} Oct 31 00:04:33.217365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.217344 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" event={"ID":"79046fb0-7fc5-4930-8416-7d9278c90ba9","Type":"ContainerStarted","Data":"635f4487aa9663ccfe4b70cf74eaea7b99ce3ed2cfcef9af445c87779e0ff9df"} Oct 31 00:04:33.266209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.266176 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:33.408462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.408390 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:33.580872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.580827 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:33.618449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.618418 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf"] Oct 31 00:04:33.621599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.621579 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.627199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.627179 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 31 00:04:33.627313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.627256 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 31 00:04:33.637764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.637746 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf"] Oct 31 00:04:33.707145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.707070 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-dbkp4\"" Oct 31 00:04:33.707145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.707070 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 31 00:04:33.748619 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.748570 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.748834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.748627 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.748920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.748827 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkhs7\" (UniqueName: \"kubernetes.io/projected/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-kube-api-access-qkhs7\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.760900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.760867 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:33.849701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.849664 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkhs7\" (UniqueName: \"kubernetes.io/projected/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-kube-api-access-qkhs7\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.849701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.849718 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.849948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.849749 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.853050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.853024 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-webhook-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.863222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.863195 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-apiservice-cert\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.863492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.863465 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkhs7\" (UniqueName: \"kubernetes.io/projected/259b6e3a-d9e6-4ca9-a59c-702a0bc17b98-kube-api-access-qkhs7\") pod \"opentelemetry-operator-webhook-6c7d65b66-zqwjf\" (UID: \"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:33.933726 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:33.933676 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:34.223284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:34.223256 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf"] Oct 31 00:04:34.226321 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:04:34.226288 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod259b6e3a_d9e6_4ca9_a59c_702a0bc17b98.slice/crio-e3a36777f9c60d3396282c3fbf9eb493c5c7038f5be07ded2db92dae059fa6fb WatchSource:0}: Error finding container e3a36777f9c60d3396282c3fbf9eb493c5c7038f5be07ded2db92dae059fa6fb: Status 404 returned error can't find the container with id e3a36777f9c60d3396282c3fbf9eb493c5c7038f5be07ded2db92dae059fa6fb Oct 31 00:04:35.228740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:35.228682 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" event={"ID":"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98","Type":"ContainerStarted","Data":"e3a36777f9c60d3396282c3fbf9eb493c5c7038f5be07ded2db92dae059fa6fb"} Oct 31 00:04:36.235288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:36.235249 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" event={"ID":"79046fb0-7fc5-4930-8416-7d9278c90ba9","Type":"ContainerStarted","Data":"679f523b85ef1d76e3e20290e902fce12b923ddf05288a2b983c294c686bb107"} Oct 31 00:04:36.258245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:36.258198 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-bgdwj" podStartSLOduration=1.5721652609999999 podStartE2EDuration="5.258180862s" totalImagesPullingTime="3.686015601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:36.257090863 +0000 UTC m=+315.248750463" watchObservedRunningTime="2026-10-31 00:04:36.258180862 +0000 UTC m=+315.249840437" Oct 31 00:04:37.241413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:37.241377 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" event={"ID":"259b6e3a-d9e6-4ca9-a59c-702a0bc17b98","Type":"ContainerStarted","Data":"8f43a942feca1188616415e06c21ef44be7223711cd6aeb97da1804349708722"} Oct 31 00:04:37.241931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:37.241474 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:37.243196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:37.243167 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" event={"ID":"2cc560d9-fc04-4e45-943a-cd074028e2fb","Type":"ContainerStarted","Data":"48f8d7911adb1a4ab72e1a5c8bf5a90ecaa83853e2b2f8bc330dae13608dd78d"} Oct 31 00:04:37.270367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:37.270303 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" podStartSLOduration=1.775199576 podStartE2EDuration="4.270283966s" totalImagesPullingTime="2.49508439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:37.268746984 +0000 UTC m=+316.260406572" watchObservedRunningTime="2026-10-31 00:04:37.270283966 +0000 UTC m=+316.261943542" Oct 31 00:04:37.298964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:37.298910 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7" podStartSLOduration=1.5532369959999999 podStartE2EDuration="5.298895024s" totalImagesPullingTime="3.745658028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:37.298296714 +0000 UTC m=+316.289956294" watchObservedRunningTime="2026-10-31 00:04:37.298895024 +0000 UTC m=+316.290554609" Oct 31 00:04:39.207477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:39.207445 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:39.390608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:39.390577 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:39.521390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:39.521358 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:39.653183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:39.653147 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:40.197624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:40.197585 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:40.339119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:40.339075 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:48.250161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:48.250129 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-6c7d65b66-zqwjf" Oct 31 00:04:54.107649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.107614 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:54.113531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.113511 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.118736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.118692 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 31 00:04:54.134878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.134851 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:54.210941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.210906 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk2jj\" (UniqueName: \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.211107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.210956 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.311397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.311361 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.311596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.311449 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk2jj\" (UniqueName: \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.314046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.314024 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.325367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.325342 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk2jj\" (UniqueName: \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\") pod \"opentelemetry-operator-controller-manager-7f7969549-ldsdw\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.425081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.424992 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:54.460144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.460107 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:54.631564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.631533 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:54.703180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:54.703058 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:54.706804 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:04:54.706776 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd9a398_9e25_4b03_812d_6a6a16a97cbd.slice/crio-00e0b0ef6595376224a097f00483b1cd2ca8dcb0130681c46b9a7aedc10736d6 WatchSource:0}: Error finding container 00e0b0ef6595376224a097f00483b1cd2ca8dcb0130681c46b9a7aedc10736d6: Status 404 returned error can't find the container with id 00e0b0ef6595376224a097f00483b1cd2ca8dcb0130681c46b9a7aedc10736d6 Oct 31 00:04:55.310835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:55.310800 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" event={"ID":"2dd9a398-9e25-4b03-812d-6a6a16a97cbd","Type":"ContainerStarted","Data":"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7"} Oct 31 00:04:55.311266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:55.310848 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" event={"ID":"2dd9a398-9e25-4b03-812d-6a6a16a97cbd","Type":"ContainerStarted","Data":"00e0b0ef6595376224a097f00483b1cd2ca8dcb0130681c46b9a7aedc10736d6"} Oct 31 00:04:55.311266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:55.310941 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:55.338750 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:55.338684 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" podStartSLOduration=1.338665004 podStartE2EDuration="1.338665004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:55.337538178 +0000 UTC m=+334.329197763" watchObservedRunningTime="2026-10-31 00:04:55.338665004 +0000 UTC m=+334.330324590" Oct 31 00:04:55.963976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:55.963939 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:56.392542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:56.392509 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:56.541504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:56.541439 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:56.789753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:56.789678 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:56.926918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:56.926885 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:57.318820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.318761 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" podUID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" containerName="manager" containerID="cri-o://a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7" gracePeriod=10 Oct 31 00:04:57.558351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.558330 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:57.637976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.637882 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\") pod \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " Oct 31 00:04:57.637976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.637957 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\" (UniqueName: \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\") pod \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\" (UID: \"2dd9a398-9e25-4b03-812d-6a6a16a97cbd\") " Oct 31 00:04:57.640356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.640327 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj" pod "2dd9a398-9e25-4b03-812d-6a6a16a97cbd" (UID: "2dd9a398-9e25-4b03-812d-6a6a16a97cbd"). InnerVolumeSpecName "kube-api-access-rk2jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:04:57.640356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.640331 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert" pod "2dd9a398-9e25-4b03-812d-6a6a16a97cbd" (UID: "2dd9a398-9e25-4b03-812d-6a6a16a97cbd"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:04:57.739038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.738997 5577 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:57.739038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.739032 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk2jj\" (UniqueName: \"kubernetes.io/projected/2dd9a398-9e25-4b03-812d-6a6a16a97cbd-kube-api-access-rk2jj\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:04:57.814917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.814884 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:57.987258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:57.987166 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:58.323539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.323500 5577 generic.go:356] "Generic (PLEG): container finished" podID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" containerID="a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7" exitCode=0 Oct 31 00:04:58.323700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.323547 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" Oct 31 00:04:58.323700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.323582 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" event={"ID":"2dd9a398-9e25-4b03-812d-6a6a16a97cbd","Type":"ContainerDied","Data":"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7"} Oct 31 00:04:58.323700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.323630 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw" event={"ID":"2dd9a398-9e25-4b03-812d-6a6a16a97cbd","Type":"ContainerDied","Data":"00e0b0ef6595376224a097f00483b1cd2ca8dcb0130681c46b9a7aedc10736d6"} Oct 31 00:04:58.323700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.323650 5577 scope.go:122] "RemoveContainer" containerID="a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7" Oct 31 00:04:58.332304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.332288 5577 scope.go:122] "RemoveContainer" containerID="a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7" Oct 31 00:04:58.332589 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:04:58.332570 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7\": container with ID starting with a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7 not found: ID does not exist" containerID="a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7" Oct 31 00:04:58.332630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.332598 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7"} err="failed to get container status \"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7\": rpc error: code = NotFound desc = could not find container \"a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7\": container with ID starting with a199c48dada4813a1c32071bc82e218c8e61e37fa8d3670e743184825c9566e7 not found: ID does not exist" Oct 31 00:04:58.346104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.346080 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:58.350943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.350921 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-7f7969549-ldsdw"] Oct 31 00:04:58.409128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.409100 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:58.542752 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:58.542691 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:04:59.129447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:04:59.129414 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" path="/var/lib/kubelet/pods/2dd9a398-9e25-4b03-812d-6a6a16a97cbd/volumes" Oct 31 00:04:59.296542 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:04:59.296507 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:05:07.398906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:07.398871 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:07.522255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:07.522199 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:07.676203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:07.676113 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:07.820358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:07.820322 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:09.168696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:09.168660 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:09.697145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:09.697108 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:09.874752 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:09.874689 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.051633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:11.051595 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.226307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:11.226277 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.542338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:11.542302 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:11.666173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:11.666139 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:12.110190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:12.110145 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:12.294910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:12.294861 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:14.121959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:14.121923 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:14.262942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:14.262905 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:16.295443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:16.295410 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:16.423570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:16.423539 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:18.507173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:18.507142 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:18.667444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:18.667414 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:24.842613 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:24.842573 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:29.291530 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:05:29.291446 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:05:39.549650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:39.549611 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:39.710197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:39.710150 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:51.245431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:51.245393 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:51.394425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:51.394391 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:52.497580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:52.497544 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:52.669062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:52.669028 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:55.677425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.677392 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:55.892634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.892567 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6"] Oct 31 00:05:55.892909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.892894 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" containerName="manager" Oct 31 00:05:55.892993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.892913 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" containerName="manager" Oct 31 00:05:55.893058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.893006 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd9a398-9e25-4b03-812d-6a6a16a97cbd" containerName="manager" Oct 31 00:05:55.896098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.896078 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:55.911327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.911301 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 31 00:05:55.911327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.911307 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 31 00:05:55.911449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.911389 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-metrics-certs\"" Oct 31 00:05:55.911449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.911391 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-dockercfg-52rkm\"" Oct 31 00:05:55.913441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.913421 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6"] Oct 31 00:05:55.996143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.996055 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfjzt\" (UniqueName: \"kubernetes.io/projected/60ca6860-a8fd-4f8c-b785-518a8f5e3542-kube-api-access-pfjzt\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:55.996143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:55.996137 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ca6860-a8fd-4f8c-b785-518a8f5e3542-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.096544 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.096515 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfjzt\" (UniqueName: \"kubernetes.io/projected/60ca6860-a8fd-4f8c-b785-518a8f5e3542-kube-api-access-pfjzt\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.096678 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.096549 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ca6860-a8fd-4f8c-b785-518a8f5e3542-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.099146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.099128 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/60ca6860-a8fd-4f8c-b785-518a8f5e3542-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.109254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.109232 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfjzt\" (UniqueName: \"kubernetes.io/projected/60ca6860-a8fd-4f8c-b785-518a8f5e3542-kube-api-access-pfjzt\") pod \"cluster-logging-operator-5d6db4c85-rwmk6\" (UID: \"60ca6860-a8fd-4f8c-b785-518a8f5e3542\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.222743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.222696 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" Oct 31 00:05:56.506936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.506905 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6"] Oct 31 00:05:56.509988 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:05:56.509957 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60ca6860_a8fd_4f8c_b785_518a8f5e3542.slice/crio-4678060e20ebc8de6236c48176bc01103b10de158850d7767a213112c1a80024 WatchSource:0}: Error finding container 4678060e20ebc8de6236c48176bc01103b10de158850d7767a213112c1a80024: Status 404 returned error can't find the container with id 4678060e20ebc8de6236c48176bc01103b10de158850d7767a213112c1a80024 Oct 31 00:05:56.544454 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:56.544394 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" event={"ID":"60ca6860-a8fd-4f8c-b785-518a8f5e3542","Type":"ContainerStarted","Data":"4678060e20ebc8de6236c48176bc01103b10de158850d7767a213112c1a80024"} Oct 31 00:05:58.300200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:58.300132 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:58.442935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:05:58.442901 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:05:59.291793 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:05:59.291756 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:06:01.565260 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:01.565223 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" event={"ID":"60ca6860-a8fd-4f8c-b785-518a8f5e3542","Type":"ContainerStarted","Data":"8a549ddb6f4177b759ed4c8ea77b420c9f0d845b2da3a052b8a1f4d2dcfd5432"} Oct 31 00:06:01.591399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:01.591349 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/cluster-logging-operator-5d6db4c85-rwmk6" podStartSLOduration=1.680547089 podStartE2EDuration="6.591336626s" totalImagesPullingTime="4.910789537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:05:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:01.591117419 +0000 UTC m=+400.582777016" watchObservedRunningTime="2026-10-31 00:06:01.591336626 +0000 UTC m=+400.582996213" Oct 31 00:06:12.487444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:12.487407 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:12.622073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:12.622039 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.407414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.407375 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.542528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.542494 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.681856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.681774 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.812531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.812501 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:16.819114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.819092 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.827352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.827326 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:16.827759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.827737 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:16.828926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.828904 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-2n88z\"" Oct 31 00:06:16.829040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.828987 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:16.835181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.835156 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:16.851547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.851527 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.851642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.851560 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.851642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.851593 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.851761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.851742 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmwrc\" (UniqueName: \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.900846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.900825 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:16.900983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.900959 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:16.900983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.900970 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:06:16.952517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.952440 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmwrc\" (UniqueName: \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.952517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.952492 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.952517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.952516 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.952865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.952686 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.953191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.953165 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.955084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.955062 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.955433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.955411 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.965190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.965169 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmwrc\" (UniqueName: \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\") pod \"data-science-collector-collector-1\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:16.974547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:16.974513 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:17.133836 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:17.133805 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:17.419132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:17.419101 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:17.422790 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:06:17.422764 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod844b9a4e_853e_4308_acf3_786f06e70c2a.slice/crio-204b63cdce24b08deb1055e3cddae0546ec114ddb2e20d6a24ee8c9c69cba46c WatchSource:0}: Error finding container 204b63cdce24b08deb1055e3cddae0546ec114ddb2e20d6a24ee8c9c69cba46c: Status 404 returned error can't find the container with id 204b63cdce24b08deb1055e3cddae0546ec114ddb2e20d6a24ee8c9c69cba46c Oct 31 00:06:17.625046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:17.625013 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"844b9a4e-853e-4308-acf3-786f06e70c2a","Type":"ContainerStarted","Data":"204b63cdce24b08deb1055e3cddae0546ec114ddb2e20d6a24ee8c9c69cba46c"} Oct 31 00:06:17.735626 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:17.735543 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:17.878064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:17.878020 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:18.002132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:18.002080 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.083184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.083150 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.217155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.217118 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.749051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.749015 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:19.751538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.751512 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.756498 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.756420 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:06:19.756693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.756668 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:06:19.762860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.762830 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:19.877440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.877400 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.877627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.877547 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqlpx\" (UniqueName: \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.877627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.877606 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.938858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.938818 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:19.979103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.979074 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqlpx\" (UniqueName: \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.979270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.979119 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.979410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.979386 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.979535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.979517 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.979951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.979932 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:19.993492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:19.993473 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqlpx\" (UniqueName: \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:20.088529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:20.088495 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:21.028213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:21.028179 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:21.774907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:21.774875 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:21.775186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:21.774934 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:22.196764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.196656 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.203867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.203847 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.298083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.298053 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.298250 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.298191 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.301823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.301801 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.475356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.475331 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.490761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.490736 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:22.650065 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.650026 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"844b9a4e-853e-4308-acf3-786f06e70c2a","Type":"ContainerStarted","Data":"d843f4f70e74af29aacf1ab4ead09fa46407b27f2cdcc8d417c8a15ec09a66bf"} Oct 31 00:06:22.673591 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.673538 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.324566787 podStartE2EDuration="6.673521319s" totalImagesPullingTime="4.348954532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:22.67209497 +0000 UTC m=+421.663754553" watchObservedRunningTime="2026-10-31 00:06:22.673521319 +0000 UTC m=+421.665180903" Oct 31 00:06:22.767909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.767874 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:06:22.776142 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:22.776124 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:23.046762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:23.046660 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:23.049419 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:06:23.049386 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1167f766_fdfb_4ea8_8760_d0daf4018172.slice/crio-b03cbef912c45d10f5c39b10c2a4851a314f20d2bf8103a627810276a6fbdece WatchSource:0}: Error finding container b03cbef912c45d10f5c39b10c2a4851a314f20d2bf8103a627810276a6fbdece: Status 404 returned error can't find the container with id b03cbef912c45d10f5c39b10c2a4851a314f20d2bf8103a627810276a6fbdece Oct 31 00:06:23.654927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:23.654888 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1167f766-fdfb-4ea8-8760-d0daf4018172","Type":"ContainerStarted","Data":"b03cbef912c45d10f5c39b10c2a4851a314f20d2bf8103a627810276a6fbdece"} Oct 31 00:06:24.884850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:24.884814 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:28.679897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:28.679801 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1167f766-fdfb-4ea8-8760-d0daf4018172","Type":"ContainerStarted","Data":"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948"} Oct 31 00:06:28.680274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:28.679939 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:28.699730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:28.699648 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=7.470439061 podStartE2EDuration="12.699607873s" totalImagesPullingTime="5.229168812s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:28.698938877 +0000 UTC m=+427.690598484" watchObservedRunningTime="2026-10-31 00:06:28.699607873 +0000 UTC m=+427.691267451" Oct 31 00:06:29.295304 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:29.295267 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:06:34.572274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.572239 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:34.572758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.572515 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="844b9a4e-853e-4308-acf3-786f06e70c2a" containerName="otc-container" containerID="cri-o://d843f4f70e74af29aacf1ab4ead09fa46407b27f2cdcc8d417c8a15ec09a66bf" gracePeriod=30 Oct 31 00:06:34.648748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.648664 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:34.671456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.671427 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:34.671617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.671595 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="1167f766-fdfb-4ea8-8760-d0daf4018172" containerName="perses" containerID="cri-o://8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948" gracePeriod=30 Oct 31 00:06:34.707902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.707862 5577 generic.go:356] "Generic (PLEG): container finished" podID="844b9a4e-853e-4308-acf3-786f06e70c2a" containerID="d843f4f70e74af29aacf1ab4ead09fa46407b27f2cdcc8d417c8a15ec09a66bf" exitCode=0 Oct 31 00:06:34.708023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.707940 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"844b9a4e-853e-4308-acf3-786f06e70c2a","Type":"ContainerDied","Data":"d843f4f70e74af29aacf1ab4ead09fa46407b27f2cdcc8d417c8a15ec09a66bf"} Oct 31 00:06:34.783927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.783865 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:34.830648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.830588 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:34.909186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.909163 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\") pod \"844b9a4e-853e-4308-acf3-786f06e70c2a\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " Oct 31 00:06:34.909316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.909258 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\") pod \"844b9a4e-853e-4308-acf3-786f06e70c2a\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " Oct 31 00:06:34.909316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.909294 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\" (UniqueName: \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\") pod \"844b9a4e-853e-4308-acf3-786f06e70c2a\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " Oct 31 00:06:34.909519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.909338 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\") pod \"844b9a4e-853e-4308-acf3-786f06e70c2a\" (UID: \"844b9a4e-853e-4308-acf3-786f06e70c2a\") " Oct 31 00:06:34.909756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.909616 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal" pod "844b9a4e-853e-4308-acf3-786f06e70c2a" (UID: "844b9a4e-853e-4308-acf3-786f06e70c2a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:34.911915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.911889 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs" pod "844b9a4e-853e-4308-acf3-786f06e70c2a" (UID: "844b9a4e-853e-4308-acf3-786f06e70c2a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:34.912018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.911998 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc" pod "844b9a4e-853e-4308-acf3-786f06e70c2a" (UID: "844b9a4e-853e-4308-acf3-786f06e70c2a"). InnerVolumeSpecName "kube-api-access-zmwrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:34.912018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.912004 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs" pod "844b9a4e-853e-4308-acf3-786f06e70c2a" (UID: "844b9a4e-853e-4308-acf3-786f06e70c2a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:34.919676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:34.919658 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:35.010114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010085 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\" (UniqueName: \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\") pod \"1167f766-fdfb-4ea8-8760-d0daf4018172\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " Oct 31 00:06:35.010251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010225 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"1167f766-fdfb-4ea8-8760-d0daf4018172\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " Oct 31 00:06:35.010292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010266 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\" (UniqueName: \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\") pod \"1167f766-fdfb-4ea8-8760-d0daf4018172\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " Oct 31 00:06:35.010323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010307 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\" (UniqueName: \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\") pod \"1167f766-fdfb-4ea8-8760-d0daf4018172\" (UID: \"1167f766-fdfb-4ea8-8760-d0daf4018172\") " Oct 31 00:06:35.010485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010470 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/844b9a4e-853e-4308-acf3-786f06e70c2a-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.010566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010491 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmwrc\" (UniqueName: \"kubernetes.io/projected/844b9a4e-853e-4308-acf3-786f06e70c2a-kube-api-access-zmwrc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.010566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010506 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.010566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010518 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/844b9a4e-853e-4308-acf3-786f06e70c2a-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.010841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.010796 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config" pod "1167f766-fdfb-4ea8-8760-d0daf4018172" (UID: "1167f766-fdfb-4ea8-8760-d0daf4018172"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:35.012572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.012550 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx" pod "1167f766-fdfb-4ea8-8760-d0daf4018172" (UID: "1167f766-fdfb-4ea8-8760-d0daf4018172"). InnerVolumeSpecName "kube-api-access-fqlpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:35.014820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.014800 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "1167f766-fdfb-4ea8-8760-d0daf4018172" (UID: "1167f766-fdfb-4ea8-8760-d0daf4018172"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:06:35.111534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.111462 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1167f766-fdfb-4ea8-8760-d0daf4018172-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.111534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.111513 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:06:35.111534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.111531 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqlpx\" (UniqueName: \"kubernetes.io/projected/1167f766-fdfb-4ea8-8760-d0daf4018172-kube-api-access-fqlpx\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.122009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.121991 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:06:35.130344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.130314 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins" pod "1167f766-fdfb-4ea8-8760-d0daf4018172" (UID: "1167f766-fdfb-4ea8-8760-d0daf4018172"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:06:35.212355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.212331 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/1167f766-fdfb-4ea8-8760-d0daf4018172-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:35.212355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.212355 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:06:35.712797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.712764 5577 generic.go:356] "Generic (PLEG): container finished" podID="1167f766-fdfb-4ea8-8760-d0daf4018172" containerID="8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948" exitCode=0 Oct 31 00:06:35.713230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.712811 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:06:35.713230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.712851 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1167f766-fdfb-4ea8-8760-d0daf4018172","Type":"ContainerDied","Data":"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948"} Oct 31 00:06:35.713230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.712895 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"1167f766-fdfb-4ea8-8760-d0daf4018172","Type":"ContainerDied","Data":"b03cbef912c45d10f5c39b10c2a4851a314f20d2bf8103a627810276a6fbdece"} Oct 31 00:06:35.713230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.712922 5577 scope.go:122] "RemoveContainer" containerID="8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948" Oct 31 00:06:35.714246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.714225 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"844b9a4e-853e-4308-acf3-786f06e70c2a","Type":"ContainerDied","Data":"204b63cdce24b08deb1055e3cddae0546ec114ddb2e20d6a24ee8c9c69cba46c"} Oct 31 00:06:35.714323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.714258 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:06:35.721973 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.721960 5577 scope.go:122] "RemoveContainer" containerID="8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948" Oct 31 00:06:35.722216 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:35.722197 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948\": container with ID starting with 8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948 not found: ID does not exist" containerID="8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948" Oct 31 00:06:35.722257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.722223 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948"} err="failed to get container status \"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948\": rpc error: code = NotFound desc = could not find container \"8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948\": container with ID starting with 8ddd261fdc00ba20ce80d25a7ef17c65209f3696b9096b7344061957da8ab948 not found: ID does not exist" Oct 31 00:06:35.722257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.722241 5577 scope.go:122] "RemoveContainer" containerID="d843f4f70e74af29aacf1ab4ead09fa46407b27f2cdcc8d417c8a15ec09a66bf" Oct 31 00:06:35.729947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.729930 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:35.734501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.734482 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:06:35.748335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.748314 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:35.753646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:35.753627 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:06:36.661565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:36.661538 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:37.129145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:37.129115 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1167f766-fdfb-4ea8-8760-d0daf4018172" path="/var/lib/kubelet/pods/1167f766-fdfb-4ea8-8760-d0daf4018172/volumes" Oct 31 00:06:37.129517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:37.129453 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="844b9a4e-853e-4308-acf3-786f06e70c2a" path="/var/lib/kubelet/pods/844b9a4e-853e-4308-acf3-786f06e70c2a/volumes" Oct 31 00:06:38.231419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.231386 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:38.365323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.365291 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:38.470320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470288 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:38.470633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470619 5577 cpu_manager.go:414] "containerMap: removing container" podUID="1167f766-fdfb-4ea8-8760-d0daf4018172" containerName="perses" Oct 31 00:06:38.470676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470635 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1167f766-fdfb-4ea8-8760-d0daf4018172" containerName="perses" Oct 31 00:06:38.470676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470647 5577 cpu_manager.go:414] "containerMap: removing container" podUID="844b9a4e-853e-4308-acf3-786f06e70c2a" containerName="otc-container" Oct 31 00:06:38.470676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470652 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="844b9a4e-853e-4308-acf3-786f06e70c2a" containerName="otc-container" Oct 31 00:06:38.470822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470732 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="1167f766-fdfb-4ea8-8760-d0daf4018172" containerName="perses" Oct 31 00:06:38.470822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.470741 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="844b9a4e-853e-4308-acf3-786f06e70c2a" containerName="otc-container" Oct 31 00:06:38.491694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.491616 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:38.491847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.491760 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.502516 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.502488 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:38.502664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.502514 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:38.503386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.503369 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:38.503386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.503369 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:06:38.503494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.503422 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-pkxt5\"" Oct 31 00:06:38.503676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.503658 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-b5f8018a\"" Oct 31 00:06:38.503790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.503777 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:38.510583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.510546 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:38.538371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.538340 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.538500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.538404 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.538564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.538496 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7zvt\" (UniqueName: \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.538564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.538543 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.538656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.538639 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.548066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.548026 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:38.551344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.551324 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.562679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.562656 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:38.605455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.605434 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:06:38.605622 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.605539 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-62j9v\"" Oct 31 00:06:38.635812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.635791 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:06:38.639035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639017 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.639102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639050 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.639102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639068 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7zvt\" (UniqueName: \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.639102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639090 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.639268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639108 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.639268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639129 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.639268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639256 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.639448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639310 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvhtp\" (UniqueName: \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.639448 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:38.639344 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:38.639448 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:38.639406 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert podName:5d951e38-7fcd-44f5-a1b2-bdbabd341afd nodeName:}" failed. No retries permitted until 2026-10-31 00:06:39.139384726 +0000 UTC m=+438.131044305 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd") : secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:38.639827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.639809 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.641975 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.641956 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.642042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.641956 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.652899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.652881 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7zvt\" (UniqueName: \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:38.657683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.657613 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:38.739967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.739932 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvhtp\" (UniqueName: \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.740145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.739985 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.740145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.740117 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.740238 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:38.740218 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-server-cert: secret "data-science-collector-ta-server-cert" not found Oct 31 00:06:38.740303 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:38.740287 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert podName:bccad32f-651e-4ca7-9f38-17b7678e5536 nodeName:}" failed. No retries permitted until 2026-10-31 00:06:39.24026884 +0000 UTC m=+438.231928419 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-server-cert" (UniqueName: "kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert") pod "data-science-collector-targetallocator-759bf9bf6f-rscfq" (UID: "bccad32f-651e-4ca7-9f38-17b7678e5536") : secret "data-science-collector-ta-server-cert" not found Oct 31 00:06:38.740508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.740490 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.755193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.755162 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvhtp\" (UniqueName: \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:38.802800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:38.802773 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:39.144090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:39.144015 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:39.144230 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:39.144152 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:39.144230 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:39.144213 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert podName:5d951e38-7fcd-44f5-a1b2-bdbabd341afd nodeName:}" failed. No retries permitted until 2026-10-31 00:06:40.144198054 +0000 UTC m=+439.135857629 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd") : secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:39.245292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:39.245255 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:39.245676 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:39.245406 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-server-cert: secret "data-science-collector-ta-server-cert" not found Oct 31 00:06:39.245676 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:39.245479 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert podName:bccad32f-651e-4ca7-9f38-17b7678e5536 nodeName:}" failed. No retries permitted until 2026-10-31 00:06:40.245460366 +0000 UTC m=+439.237119945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-server-cert" (UniqueName: "kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert") pod "data-science-collector-targetallocator-759bf9bf6f-rscfq" (UID: "bccad32f-651e-4ca7-9f38-17b7678e5536") : secret "data-science-collector-ta-server-cert" not found Oct 31 00:06:40.153020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.152982 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:40.153196 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:40.153121 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:40.153196 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:40.153173 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert podName:5d951e38-7fcd-44f5-a1b2-bdbabd341afd nodeName:}" failed. No retries permitted until 2026-10-31 00:06:42.153161224 +0000 UTC m=+441.144820786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert") pod "data-science-collector-collector-0" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd") : secret "data-science-collector-ta-client-cert" not found Oct 31 00:06:40.254123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.254091 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:40.256848 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.256826 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-759bf9bf6f-rscfq\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:40.363002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.362970 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:40.637106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.637069 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:40.640884 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:06:40.640857 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbccad32f_651e_4ca7_9f38_17b7678e5536.slice/crio-9573846e810cb9e8a1e086aa3a093f02f15c9af22e6e194e7d2666d1f3cafce0 WatchSource:0}: Error finding container 9573846e810cb9e8a1e086aa3a093f02f15c9af22e6e194e7d2666d1f3cafce0: Status 404 returned error can't find the container with id 9573846e810cb9e8a1e086aa3a093f02f15c9af22e6e194e7d2666d1f3cafce0 Oct 31 00:06:40.737125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:40.737056 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" event={"ID":"bccad32f-651e-4ca7-9f38-17b7678e5536","Type":"ContainerStarted","Data":"9573846e810cb9e8a1e086aa3a093f02f15c9af22e6e194e7d2666d1f3cafce0"} Oct 31 00:06:42.169490 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.169453 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:42.171952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.171928 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:42.406422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.406389 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:42.684327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.684289 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:42.687317 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:06:42.687289 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d951e38_7fcd_44f5_a1b2_bdbabd341afd.slice/crio-88f40b7740cce3c2ca96c95cde9d886dfb4f6a798c9c5278f00512273862eb8b WatchSource:0}: Error finding container 88f40b7740cce3c2ca96c95cde9d886dfb4f6a798c9c5278f00512273862eb8b: Status 404 returned error can't find the container with id 88f40b7740cce3c2ca96c95cde9d886dfb4f6a798c9c5278f00512273862eb8b Oct 31 00:06:42.690781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.690750 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:42.690834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.690813 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:42.745726 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:42.745685 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d951e38-7fcd-44f5-a1b2-bdbabd341afd","Type":"ContainerStarted","Data":"88f40b7740cce3c2ca96c95cde9d886dfb4f6a798c9c5278f00512273862eb8b"} Oct 31 00:06:43.752227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:43.752184 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d951e38-7fcd-44f5-a1b2-bdbabd341afd","Type":"ContainerStarted","Data":"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04"} Oct 31 00:06:43.774006 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:43.773935 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=5.773921543 podStartE2EDuration="5.773921543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:43.773146047 +0000 UTC m=+442.764805633" watchObservedRunningTime="2026-10-31 00:06:43.773921543 +0000 UTC m=+442.765581123" Oct 31 00:06:44.404361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.404333 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:44.404442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.404393 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:44.757955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.757920 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" event={"ID":"bccad32f-651e-4ca7-9f38-17b7678e5536","Type":"ContainerStarted","Data":"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0"} Oct 31 00:06:44.758424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.758062 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:44.759749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.759730 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:44.778184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:44.778136 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" podStartSLOduration=3.017342361 podStartE2EDuration="6.778121313s" totalImagesPullingTime="3.760778952s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:44.778035167 +0000 UTC m=+443.769694752" watchObservedRunningTime="2026-10-31 00:06:44.778121313 +0000 UTC m=+443.769780902" Oct 31 00:06:48.601285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:48.601245 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:52.045869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.045834 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:52.046328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.046062 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" containerName="otc-container" containerID="cri-o://398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04" gracePeriod=30 Oct 31 00:06:52.086925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.086892 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:52.087093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.087073 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" podUID="bccad32f-651e-4ca7-9f38-17b7678e5536" containerName="ta-container" containerID="cri-o://49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0" gracePeriod=30 Oct 31 00:06:52.120620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.120586 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:52.254380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.254355 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:52.354512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.354491 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:52.357102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.357083 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:52.460895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.460868 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\") pod \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " Oct 31 00:06:52.461067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.460909 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\" (UniqueName: \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\") pod \"bccad32f-651e-4ca7-9f38-17b7678e5536\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " Oct 31 00:06:52.461067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.460951 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\") pod \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " Oct 31 00:06:52.461067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461020 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\" (UniqueName: \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\") pod \"bccad32f-651e-4ca7-9f38-17b7678e5536\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " Oct 31 00:06:52.461239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461056 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\") pod \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " Oct 31 00:06:52.461239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461129 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\" (UniqueName: \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\") pod \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " Oct 31 00:06:52.461239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461156 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") pod \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\" (UID: \"5d951e38-7fcd-44f5-a1b2-bdbabd341afd\") " Oct 31 00:06:52.461239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461191 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") pod \"bccad32f-651e-4ca7-9f38-17b7678e5536\" (UID: \"bccad32f-651e-4ca7-9f38-17b7678e5536\") " Oct 31 00:06:52.461444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461294 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal" pod "bccad32f-651e-4ca7-9f38-17b7678e5536" (UID: "bccad32f-651e-4ca7-9f38-17b7678e5536"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:52.461444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461419 5577 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/bccad32f-651e-4ca7-9f38-17b7678e5536-ta-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.461557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.461509 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal" pod "5d951e38-7fcd-44f5-a1b2-bdbabd341afd" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:06:52.463687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.463657 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert" pod "bccad32f-651e-4ca7-9f38-17b7678e5536" (UID: "bccad32f-651e-4ca7-9f38-17b7678e5536"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.463841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.463700 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert" pod "5d951e38-7fcd-44f5-a1b2-bdbabd341afd" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.463841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.463797 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs" pod "5d951e38-7fcd-44f5-a1b2-bdbabd341afd" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.463945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.463851 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt" pod "5d951e38-7fcd-44f5-a1b2-bdbabd341afd" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd"). InnerVolumeSpecName "kube-api-access-m7zvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:52.464094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.464068 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp" pod "bccad32f-651e-4ca7-9f38-17b7678e5536" (UID: "bccad32f-651e-4ca7-9f38-17b7678e5536"). InnerVolumeSpecName "kube-api-access-fvhtp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:06:52.464094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.464080 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs" pod "5d951e38-7fcd-44f5-a1b2-bdbabd341afd" (UID: "5d951e38-7fcd-44f5-a1b2-bdbabd341afd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:06:52.562004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.561971 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7zvt\" (UniqueName: \"kubernetes.io/projected/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-kube-api-access-m7zvt\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.561999 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.562012 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/bccad32f-651e-4ca7-9f38-17b7678e5536-data-science-collector-ta-server-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.562022 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.562031 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.562039 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvhtp\" (UniqueName: \"kubernetes.io/projected/bccad32f-651e-4ca7-9f38-17b7678e5536-kube-api-access-fvhtp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.562246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.562048 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d951e38-7fcd-44f5-a1b2-bdbabd341afd-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:06:52.801199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.801165 5577 generic.go:356] "Generic (PLEG): container finished" podID="bccad32f-651e-4ca7-9f38-17b7678e5536" containerID="49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0" exitCode=0 Oct 31 00:06:52.801409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.801219 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" Oct 31 00:06:52.801409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.801262 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" event={"ID":"bccad32f-651e-4ca7-9f38-17b7678e5536","Type":"ContainerDied","Data":"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0"} Oct 31 00:06:52.801409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.801309 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq" event={"ID":"bccad32f-651e-4ca7-9f38-17b7678e5536","Type":"ContainerDied","Data":"9573846e810cb9e8a1e086aa3a093f02f15c9af22e6e194e7d2666d1f3cafce0"} Oct 31 00:06:52.801409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.801330 5577 scope.go:122] "RemoveContainer" containerID="49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0" Oct 31 00:06:52.803161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.803053 5577 generic.go:356] "Generic (PLEG): container finished" podID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" containerID="398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04" exitCode=0 Oct 31 00:06:52.803250 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.803110 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:52.803332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.803131 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d951e38-7fcd-44f5-a1b2-bdbabd341afd","Type":"ContainerDied","Data":"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04"} Oct 31 00:06:52.803413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.803397 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5d951e38-7fcd-44f5-a1b2-bdbabd341afd","Type":"ContainerDied","Data":"88f40b7740cce3c2ca96c95cde9d886dfb4f6a798c9c5278f00512273862eb8b"} Oct 31 00:06:52.810500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.810420 5577 scope.go:122] "RemoveContainer" containerID="49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0" Oct 31 00:06:52.810819 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:52.810792 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0\": container with ID starting with 49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0 not found: ID does not exist" containerID="49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0" Oct 31 00:06:52.810921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.810824 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0"} err="failed to get container status \"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0\": rpc error: code = NotFound desc = could not find container \"49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0\": container with ID starting with 49fb68296a8e37fbf3f3e75678cc6a2ca244033a8dce83878371b9b80c2286a0 not found: ID does not exist" Oct 31 00:06:52.810921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.810840 5577 scope.go:122] "RemoveContainer" containerID="398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04" Oct 31 00:06:52.818439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.818418 5577 scope.go:122] "RemoveContainer" containerID="398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04" Oct 31 00:06:52.818692 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:52.818676 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04\": container with ID starting with 398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04 not found: ID does not exist" containerID="398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04" Oct 31 00:06:52.819351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.818700 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04"} err="failed to get container status \"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04\": rpc error: code = NotFound desc = could not find container \"398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04\": container with ID starting with 398f0152ea89e622666337b9f9cd8d6acb22b2088cf93fd346be7835d18d2b04 not found: ID does not exist" Oct 31 00:06:52.826096 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.826076 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:52.830632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.830613 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:52.844065 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.844041 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:52.851527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:52.851508 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-759bf9bf6f-rscfq"] Oct 31 00:06:53.128357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:53.128325 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" path="/var/lib/kubelet/pods/5d951e38-7fcd-44f5-a1b2-bdbabd341afd/volumes" Oct 31 00:06:53.128739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:53.128678 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bccad32f-651e-4ca7-9f38-17b7678e5536" path="/var/lib/kubelet/pods/bccad32f-651e-4ca7-9f38-17b7678e5536/volumes" Oct 31 00:06:54.305696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.305665 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.433342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.433304 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.489334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489307 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:54.489610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489599 5577 cpu_manager.go:414] "containerMap: removing container" podUID="bccad32f-651e-4ca7-9f38-17b7678e5536" containerName="ta-container" Oct 31 00:06:54.489654 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489612 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bccad32f-651e-4ca7-9f38-17b7678e5536" containerName="ta-container" Oct 31 00:06:54.489654 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489636 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" containerName="otc-container" Oct 31 00:06:54.489654 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489641 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" containerName="otc-container" Oct 31 00:06:54.489766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489688 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="bccad32f-651e-4ca7-9f38-17b7678e5536" containerName="ta-container" Oct 31 00:06:54.489766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.489698 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d951e38-7fcd-44f5-a1b2-bdbabd341afd" containerName="otc-container" Oct 31 00:06:54.538628 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.538592 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:54.538793 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.538687 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.570350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.570280 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.617964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.617940 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:06:54.618108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.617991 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:06:54.618108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.618031 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:06:54.618227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.618108 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-c631c892\"" Oct 31 00:06:54.618227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.618168 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:06:54.675893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.675854 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.675893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.675897 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.676085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.675921 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.676085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.676010 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwnhr\" (UniqueName: \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.704155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.704118 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:54.776893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.776868 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.777038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.776897 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.777038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.776922 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.777038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.776939 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwnhr\" (UniqueName: \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.777534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.777513 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.779529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.779509 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.779612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.779551 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.789509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.789491 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwnhr\" (UniqueName: \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\") pod \"data-science-collector-collector-0\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.862436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.862365 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:06:54.876600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:54.876571 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:06:55.134964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.134896 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:06:55.138288 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:06:55.138247 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod782894de_cb70_4792_8ddd_7003ee0d654a.slice/crio-ccbc9485ee7a19eec3b5072cbad5e03ffa6dbd390c3d406530d7f7fa0c134f65 WatchSource:0}: Error finding container ccbc9485ee7a19eec3b5072cbad5e03ffa6dbd390c3d406530d7f7fa0c134f65: Status 404 returned error can't find the container with id ccbc9485ee7a19eec3b5072cbad5e03ffa6dbd390c3d406530d7f7fa0c134f65 Oct 31 00:06:55.140438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.140402 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:55.140517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.140475 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:06:55.818445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.818413 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"782894de-cb70-4792-8ddd-7003ee0d654a","Type":"ContainerStarted","Data":"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275"} Oct 31 00:06:55.818824 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.818460 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"782894de-cb70-4792-8ddd-7003ee0d654a","Type":"ContainerStarted","Data":"ccbc9485ee7a19eec3b5072cbad5e03ffa6dbd390c3d406530d7f7fa0c134f65"} Oct 31 00:06:55.838371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:06:55.838311 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.838296931 podStartE2EDuration="1.838296931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:06:55.837392657 +0000 UTC m=+454.829052242" watchObservedRunningTime="2026-10-31 00:06:55.838296931 +0000 UTC m=+454.829956508" Oct 31 00:06:59.290841 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:06:59.290761 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:07:08.481782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.481748 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:08.482316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.482022 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="782894de-cb70-4792-8ddd-7003ee0d654a" containerName="otc-container" containerID="cri-o://dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275" gracePeriod=30 Oct 31 00:07:08.581784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.581746 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:08.712169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.712145 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:08.712594 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.712571 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:08.871684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.871645 5577 generic.go:356] "Generic (PLEG): container finished" podID="782894de-cb70-4792-8ddd-7003ee0d654a" containerID="dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275" exitCode=0 Oct 31 00:07:08.871877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.871722 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:08.871877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.871738 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"782894de-cb70-4792-8ddd-7003ee0d654a","Type":"ContainerDied","Data":"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275"} Oct 31 00:07:08.871877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.871803 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"782894de-cb70-4792-8ddd-7003ee0d654a","Type":"ContainerDied","Data":"ccbc9485ee7a19eec3b5072cbad5e03ffa6dbd390c3d406530d7f7fa0c134f65"} Oct 31 00:07:08.871877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.871832 5577 scope.go:122] "RemoveContainer" containerID="dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275" Oct 31 00:07:08.880046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880028 5577 scope.go:122] "RemoveContainer" containerID="dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275" Oct 31 00:07:08.880193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880170 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\" (UniqueName: \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\") pod \"782894de-cb70-4792-8ddd-7003ee0d654a\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " Oct 31 00:07:08.880313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880232 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\") pod \"782894de-cb70-4792-8ddd-7003ee0d654a\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " Oct 31 00:07:08.880313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880257 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\") pod \"782894de-cb70-4792-8ddd-7003ee0d654a\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " Oct 31 00:07:08.880313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880299 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\") pod \"782894de-cb70-4792-8ddd-7003ee0d654a\" (UID: \"782894de-cb70-4792-8ddd-7003ee0d654a\") " Oct 31 00:07:08.880484 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:08.880402 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275\": container with ID starting with dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275 not found: ID does not exist" containerID="dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275" Oct 31 00:07:08.880484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880431 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275"} err="failed to get container status \"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275\": rpc error: code = NotFound desc = could not find container \"dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275\": container with ID starting with dd0fae28f9419ab00cdb1c1462eafd372897caf158714efa99a6012980647275 not found: ID does not exist" Oct 31 00:07:08.881148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.880691 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal" pod "782894de-cb70-4792-8ddd-7003ee0d654a" (UID: "782894de-cb70-4792-8ddd-7003ee0d654a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:08.882421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.882386 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs" pod "782894de-cb70-4792-8ddd-7003ee0d654a" (UID: "782894de-cb70-4792-8ddd-7003ee0d654a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:08.882547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.882530 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr" pod "782894de-cb70-4792-8ddd-7003ee0d654a" (UID: "782894de-cb70-4792-8ddd-7003ee0d654a"). InnerVolumeSpecName "kube-api-access-jwnhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:08.882590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.882529 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs" pod "782894de-cb70-4792-8ddd-7003ee0d654a" (UID: "782894de-cb70-4792-8ddd-7003ee0d654a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:08.981702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.981664 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.981702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.981695 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/782894de-cb70-4792-8ddd-7003ee0d654a-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.981702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.981728 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/782894de-cb70-4792-8ddd-7003ee0d654a-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:08.981957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:08.981737 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwnhr\" (UniqueName: \"kubernetes.io/projected/782894de-cb70-4792-8ddd-7003ee0d654a-kube-api-access-jwnhr\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:09.187335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:09.187302 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:09.192114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:09.192087 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:11.129183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:11.129148 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="782894de-cb70-4792-8ddd-7003ee0d654a" path="/var/lib/kubelet/pods/782894de-cb70-4792-8ddd-7003ee0d654a/volumes" Oct 31 00:07:11.822931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:11.822900 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:11.950496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:11.950428 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:12.100677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.100601 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:12.101095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.101080 5577 cpu_manager.go:414] "containerMap: removing container" podUID="782894de-cb70-4792-8ddd-7003ee0d654a" containerName="otc-container" Oct 31 00:07:12.101138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.101100 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="782894de-cb70-4792-8ddd-7003ee0d654a" containerName="otc-container" Oct 31 00:07:12.101174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.101163 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="782894de-cb70-4792-8ddd-7003ee0d654a" containerName="otc-container" Oct 31 00:07:12.108277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.108248 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.115156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.114896 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:07:12.115156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.115042 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:12.118926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.117790 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:12.118926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.117892 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:07:12.118926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.117945 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:07:12.118926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.118049 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:12.118926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.118134 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4d267ad2\"" Oct 31 00:07:12.190638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.190608 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:12.193247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.193225 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.193402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.193284 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:12.202615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.202455 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:12.208947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.208922 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.209051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.208963 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.209051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.208984 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxfk4\" (UniqueName: \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.209134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.209106 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.209171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.209140 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.214271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.214249 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:07:12.214417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.214399 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-86flm\"" Oct 31 00:07:12.214477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.214407 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:07:12.309883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.309855 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.310071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.309897 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.310071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.309955 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.310071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.309992 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.310071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.310023 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.310272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.310199 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c75g4\" (UniqueName: \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.310272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.310245 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.310375 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.310272 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxfk4\" (UniqueName: \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.310953 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.310927 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.312595 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.312568 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.312736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.312616 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.312736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.312653 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.320194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.320163 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:12.322496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.322474 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxfk4\" (UniqueName: \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\") pod \"data-science-collector-collector-0\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.411443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.411361 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.411443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.411402 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c75g4\" (UniqueName: \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.411631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.411450 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.412056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.412039 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.414124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.414100 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.423876 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.423853 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c75g4\" (UniqueName: \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\") pod \"data-science-collector-targetallocator-7c5b7d86f-88fkt\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.424570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.424551 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:12.503944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.503916 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.699734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.699631 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:12.705087 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:12.705057 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad8bc0b3_af25_4c09_9a41_c2b8011f4fbd.slice/crio-c45f455c1855039c56493c89fddb7c95a4fc1dc72032b24835f594a6f6ca5e5a WatchSource:0}: Error finding container c45f455c1855039c56493c89fddb7c95a4fc1dc72032b24835f594a6f6ca5e5a: Status 404 returned error can't find the container with id c45f455c1855039c56493c89fddb7c95a4fc1dc72032b24835f594a6f6ca5e5a Oct 31 00:07:12.707484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.707456 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:12.707548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.707509 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:12.774864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.774837 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:12.778066 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:12.778033 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5292cc96_1497_43a4_9400_19f938843f0f.slice/crio-a22c616bc7e821edf28953878aef3a4e011f02d4bc30bf9b06d27bc775a3f865 WatchSource:0}: Error finding container a22c616bc7e821edf28953878aef3a4e011f02d4bc30bf9b06d27bc775a3f865: Status 404 returned error can't find the container with id a22c616bc7e821edf28953878aef3a4e011f02d4bc30bf9b06d27bc775a3f865 Oct 31 00:07:12.780099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.780070 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:12.780197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.780139 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:12.892694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.892657 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" event={"ID":"5292cc96-1497-43a4-9400-19f938843f0f","Type":"ContainerStarted","Data":"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019"} Oct 31 00:07:12.892907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.892728 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" event={"ID":"5292cc96-1497-43a4-9400-19f938843f0f","Type":"ContainerStarted","Data":"a22c616bc7e821edf28953878aef3a4e011f02d4bc30bf9b06d27bc775a3f865"} Oct 31 00:07:12.893032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.893013 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:12.894503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.894462 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" podUID="5292cc96-1497-43a4-9400-19f938843f0f" containerName="ta-container" probeResult="failure" output="Get \"http://10.133.0.26:8080/readyz\": dial tcp 10.133.0.26:8080: connect: connection refused" Oct 31 00:07:12.894777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.894752 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd","Type":"ContainerStarted","Data":"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5"} Oct 31 00:07:12.894851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.894802 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd","Type":"ContainerStarted","Data":"c45f455c1855039c56493c89fddb7c95a4fc1dc72032b24835f594a6f6ca5e5a"} Oct 31 00:07:12.911320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.911268 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" podStartSLOduration=0.911249467 podStartE2EDuration="911.249467ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:12.910846846 +0000 UTC m=+471.902506430" watchObservedRunningTime="2026-10-31 00:07:12.911249467 +0000 UTC m=+471.902909051" Oct 31 00:07:12.931561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:12.931513 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.931498625 podStartE2EDuration="931.498625ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:12.930515822 +0000 UTC m=+471.922175474" watchObservedRunningTime="2026-10-31 00:07:12.931498625 +0000 UTC m=+471.923158205" Oct 31 00:07:13.899642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:13.899614 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:24.947566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:24.947518 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:25.442257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.442222 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:25.442774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.442739 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" containerName="otc-container" containerID="cri-o://80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5" gracePeriod=30 Oct 31 00:07:25.483858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.483829 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:25.484069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.484034 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" podUID="5292cc96-1497-43a4-9400-19f938843f0f" containerName="ta-container" containerID="cri-o://93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019" gracePeriod=30 Oct 31 00:07:25.533971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.533919 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:25.675999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.675961 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:25.746334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.746310 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:25.748889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.748870 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:25.818970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.818943 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\" (UniqueName: \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\") pod \"5292cc96-1497-43a4-9400-19f938843f0f\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.818993 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\") pod \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819011 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\" (UniqueName: \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\") pod \"5292cc96-1497-43a4-9400-19f938843f0f\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819057 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\") pod \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819087 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\") pod \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819107 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\") pod \"5292cc96-1497-43a4-9400-19f938843f0f\" (UID: \"5292cc96-1497-43a4-9400-19f938843f0f\") " Oct 31 00:07:25.819169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819139 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\") pod \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " Oct 31 00:07:25.819488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819195 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\" (UniqueName: \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\") pod \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\" (UID: \"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd\") " Oct 31 00:07:25.819548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819451 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal" pod "5292cc96-1497-43a4-9400-19f938843f0f" (UID: "5292cc96-1497-43a4-9400-19f938843f0f"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:25.819660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.819627 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal" pod "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" (UID: "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:07:25.821413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.821368 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert" pod "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" (UID: "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.821533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.821482 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs" pod "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" (UID: "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.821808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.821786 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert" pod "5292cc96-1497-43a4-9400-19f938843f0f" (UID: "5292cc96-1497-43a4-9400-19f938843f0f"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.821808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.821800 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4" pod "5292cc96-1497-43a4-9400-19f938843f0f" (UID: "5292cc96-1497-43a4-9400-19f938843f0f"). InnerVolumeSpecName "kube-api-access-c75g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:25.822005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.821990 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4" pod "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" (UID: "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd"). InnerVolumeSpecName "kube-api-access-sxfk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:07:25.822078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.822034 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs" pod "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" (UID: "ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:07:25.920611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920575 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920603 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920613 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/5292cc96-1497-43a4-9400-19f938843f0f-data-science-collector-ta-server-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920622 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920634 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxfk4\" (UniqueName: \"kubernetes.io/projected/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-kube-api-access-sxfk4\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920643 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c75g4\" (UniqueName: \"kubernetes.io/projected/5292cc96-1497-43a4-9400-19f938843f0f-kube-api-access-c75g4\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920653 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.920858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.920661 5577 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/5292cc96-1497-43a4-9400-19f938843f0f-ta-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:07:25.947962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.947933 5577 generic.go:356] "Generic (PLEG): container finished" podID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" containerID="80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5" exitCode=0 Oct 31 00:07:25.948447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.947989 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:25.948447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.948035 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd","Type":"ContainerDied","Data":"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5"} Oct 31 00:07:25.948447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.948103 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd","Type":"ContainerDied","Data":"c45f455c1855039c56493c89fddb7c95a4fc1dc72032b24835f594a6f6ca5e5a"} Oct 31 00:07:25.948447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.948138 5577 scope.go:122] "RemoveContainer" containerID="80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5" Oct 31 00:07:25.949361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.949339 5577 generic.go:356] "Generic (PLEG): container finished" podID="5292cc96-1497-43a4-9400-19f938843f0f" containerID="93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019" exitCode=0 Oct 31 00:07:25.949445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.949377 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" Oct 31 00:07:25.949445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.949397 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" event={"ID":"5292cc96-1497-43a4-9400-19f938843f0f","Type":"ContainerDied","Data":"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019"} Oct 31 00:07:25.949445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.949423 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt" event={"ID":"5292cc96-1497-43a4-9400-19f938843f0f","Type":"ContainerDied","Data":"a22c616bc7e821edf28953878aef3a4e011f02d4bc30bf9b06d27bc775a3f865"} Oct 31 00:07:25.956154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.956124 5577 scope.go:122] "RemoveContainer" containerID="80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5" Oct 31 00:07:25.956426 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:25.956408 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5\": container with ID starting with 80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5 not found: ID does not exist" containerID="80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5" Oct 31 00:07:25.956497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.956434 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5"} err="failed to get container status \"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5\": rpc error: code = NotFound desc = could not find container \"80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5\": container with ID starting with 80aa0ae321503bc3f49943c4386dc84050bec8ce0b4351b464184029e11b61f5 not found: ID does not exist" Oct 31 00:07:25.956497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.956450 5577 scope.go:122] "RemoveContainer" containerID="93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019" Oct 31 00:07:25.963962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.963946 5577 scope.go:122] "RemoveContainer" containerID="93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019" Oct 31 00:07:25.964228 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:25.964210 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019\": container with ID starting with 93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019 not found: ID does not exist" containerID="93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019" Oct 31 00:07:25.964275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.964234 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019"} err="failed to get container status \"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019\": rpc error: code = NotFound desc = could not find container \"93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019\": container with ID starting with 93dda5362bf76c335c041764a60ed59b2f1f385a655f4614d570bc53b6f85019 not found: ID does not exist" Oct 31 00:07:25.971157 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.971136 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:25.977835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.977792 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-7c5b7d86f-88fkt"] Oct 31 00:07:25.989440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.989421 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:25.993929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:25.993912 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:27.129268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:27.129233 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5292cc96-1497-43a4-9400-19f938843f0f" path="/var/lib/kubelet/pods/5292cc96-1497-43a4-9400-19f938843f0f/volumes" Oct 31 00:07:27.129745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:27.129577 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" path="/var/lib/kubelet/pods/ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd/volumes" Oct 31 00:07:29.293880 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:29.293847 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:07:32.982339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:32.982301 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.107768 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.107741 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:07:33.108066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108053 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" containerName="otc-container" Oct 31 00:07:33.108125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108067 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" containerName="otc-container" Oct 31 00:07:33.108125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108094 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5292cc96-1497-43a4-9400-19f938843f0f" containerName="ta-container" Oct 31 00:07:33.108125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108100 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5292cc96-1497-43a4-9400-19f938843f0f" containerName="ta-container" Oct 31 00:07:33.108230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108144 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad8bc0b3-af25-4c09-9a41-c2b8011f4fbd" containerName="otc-container" Oct 31 00:07:33.108230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.108156 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5292cc96-1497-43a4-9400-19f938843f0f" containerName="ta-container" Oct 31 00:07:33.111249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.111234 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.131195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.131171 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:07:33.151703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.151668 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:07:33.155073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.155047 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.168434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.168404 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:07:33.174840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.174816 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.174961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.174849 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.174961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.174866 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.175064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.174951 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfpw6\" (UniqueName: \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.175064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.174991 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.175064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.175049 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.175154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.175135 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.175218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.175199 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42lcc\" (UniqueName: \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.179530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.179503 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.187014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.186972 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:07:33.187014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.187006 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:07:33.187014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.186974 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:07:33.187194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.187001 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:07:33.187194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.187013 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:07:33.187194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.186991 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:07:33.188137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.188110 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:07:33.188252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.188161 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-t7v6l\"" Oct 31 00:07:33.188562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.188534 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:07:33.188656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.188568 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-cch9n\"" Oct 31 00:07:33.275751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275722 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42lcc\" (UniqueName: \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.275907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275770 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.275907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275799 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.276029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275913 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275943 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfpw6\" (UniqueName: \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.275973 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.276029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276002 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276067 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276226 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.276098 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:07:33.276226 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.276166 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls podName:66e9802d-8e7d-4a37-8352-a7f05b60ada6 nodeName:}" failed. No retries permitted until 2026-10-31 00:07:33.776146735 +0000 UTC m=+492.767806316 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-pqnnn" (UID: "66e9802d-8e7d-4a37-8352-a7f05b60ada6") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:07:33.276514 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276493 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276509 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.276576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276536 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.276669 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.276654 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.278686 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.278666 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.288566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.288543 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42lcc\" (UniqueName: \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.289361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.289343 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfpw6\" (UniqueName: \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-6bkth\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.317510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.317481 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.428524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.428492 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:07:33.451722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.451680 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.615327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.615245 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.642033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.642002 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:33.646317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.646298 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.655977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.655956 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:07:33.656480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.656396 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:07:33.657150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.656654 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:07:33.657253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.657198 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:07:33.659992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.659968 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:33.662128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.662106 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:07:33.678672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.678648 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:07:33.679985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.679965 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.680165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.679995 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqvrb\" (UniqueName: \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.680165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.680057 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.680165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.680080 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.680165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.680104 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.680319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.680267 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.681954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.681934 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.694651 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.694624 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:07:33.713398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.713332 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:07:33.716811 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:33.716789 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc842037b_4674_4374_8b9d_2f21d06b80d4.slice/crio-281c9d8543cb79c914c7f9b984b7b157cd333a01e853cb2380ca2f5ea4e5784f WatchSource:0}: Error finding container 281c9d8543cb79c914c7f9b984b7b157cd333a01e853cb2380ca2f5ea4e5784f: Status 404 returned error can't find the container with id 281c9d8543cb79c914c7f9b984b7b157cd333a01e853cb2380ca2f5ea4e5784f Oct 31 00:07:33.756069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.756050 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:07:33.757949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.757932 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:07:33.781471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781445 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.781562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781486 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.781562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781517 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.781629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781557 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.781676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781661 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqvrb\" (UniqueName: \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.781745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781722 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pml9m\" (UniqueName: \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.781795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781767 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:33.781849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781797 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.781897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781859 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.781948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.781916 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.782003 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.781917 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:07:33.782056 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.781958 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:07:33.782056 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.782051 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls podName:66e9802d-8e7d-4a37-8352-a7f05b60ada6 nodeName:}" failed. No retries permitted until 2026-10-31 00:07:34.782032516 +0000 UTC m=+493.773692093 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-pqnnn" (UID: "66e9802d-8e7d-4a37-8352-a7f05b60ada6") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:07:33.782161 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:33.782113 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs podName:d7d730e6-b4af-4498-a2ed-94e5346a27b1 nodeName:}" failed. No retries permitted until 2026-10-31 00:07:34.282091965 +0000 UTC m=+493.273751530 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:07:33.782271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.782234 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.784098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.784077 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.784338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.784319 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.784376 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.784339 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.790353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.790327 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.799369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.799351 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqvrb\" (UniqueName: \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:33.882630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.882572 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.882764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.882633 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.882764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.882661 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pml9m\" (UniqueName: \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.883045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.883028 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.883154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.883138 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.895479 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.895461 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pml9m\" (UniqueName: \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:33.926984 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.926947 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:33.984786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:33.984757 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerStarted","Data":"281c9d8543cb79c914c7f9b984b7b157cd333a01e853cb2380ca2f5ea4e5784f"} Oct 31 00:07:34.248363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.248273 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:34.287087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.287047 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:34.290021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.289995 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:34.382219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.382182 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:34.518102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.518062 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:34.563653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.563623 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:07:34.657033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.656969 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:34.791173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.791080 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:34.798472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.798434 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pqnnn\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:34.851476 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.851442 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:07:34.855873 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:34.855845 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d730e6_b4af_4498_a2ed_94e5346a27b1.slice/crio-681dbbef670c5d8057f28531b33e0298b9e845450fa5ce64ad66641aab25062c WatchSource:0}: Error finding container 681dbbef670c5d8057f28531b33e0298b9e845450fa5ce64ad66641aab25062c: Status 404 returned error can't find the container with id 681dbbef670c5d8057f28531b33e0298b9e845450fa5ce64ad66641aab25062c Oct 31 00:07:34.859339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.859294 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:34.859448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.859377 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:07:34.973863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.973824 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:07:34.995168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.995139 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d7d730e6-b4af-4498-a2ed-94e5346a27b1","Type":"ContainerStarted","Data":"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79"} Oct 31 00:07:34.995653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:34.995203 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d7d730e6-b4af-4498-a2ed-94e5346a27b1","Type":"ContainerStarted","Data":"681dbbef670c5d8057f28531b33e0298b9e845450fa5ce64ad66641aab25062c"} Oct 31 00:07:35.019338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:35.018756 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.018732265 podStartE2EDuration="2.018732265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:35.017521935 +0000 UTC m=+494.009181528" watchObservedRunningTime="2026-10-31 00:07:35.018732265 +0000 UTC m=+494.010391855" Oct 31 00:07:35.281204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:35.281171 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:07:35.285126 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:35.285079 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66e9802d_8e7d_4a37_8352_a7f05b60ada6.slice/crio-da369674a4db7663b9333cefd7a1ad1664ac952818d55a9603648b98554863e2 WatchSource:0}: Error finding container da369674a4db7663b9333cefd7a1ad1664ac952818d55a9603648b98554863e2: Status 404 returned error can't find the container with id da369674a4db7663b9333cefd7a1ad1664ac952818d55a9603648b98554863e2 Oct 31 00:07:36.006511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:36.006419 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerStarted","Data":"da369674a4db7663b9333cefd7a1ad1664ac952818d55a9603648b98554863e2"} Oct 31 00:07:37.016515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:37.016488 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerStarted","Data":"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70"} Oct 31 00:07:37.016839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:37.016565 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerStarted","Data":"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4"} Oct 31 00:07:38.022857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:38.022825 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerStarted","Data":"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd"} Oct 31 00:07:38.023231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:38.022897 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerStarted","Data":"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e"} Oct 31 00:07:39.028532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:39.028488 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerStarted","Data":"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633"} Oct 31 00:07:39.030592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:39.030566 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerStarted","Data":"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3"} Oct 31 00:07:39.050793 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:39.050725 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" podStartSLOduration=2.908901332 podStartE2EDuration="6.050691319s" totalImagesPullingTime="3.141789987s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:39.049163704 +0000 UTC m=+498.040823344" watchObservedRunningTime="2026-10-31 00:07:39.050691319 +0000 UTC m=+498.042350902" Oct 31 00:07:39.071057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:39.070997 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" podStartSLOduration=1.402919024 podStartE2EDuration="6.070979651s" totalImagesPullingTime="4.668060627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:39.069952538 +0000 UTC m=+498.061612156" watchObservedRunningTime="2026-10-31 00:07:39.070979651 +0000 UTC m=+498.062639236" Oct 31 00:07:40.042140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.042110 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.270075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.270042 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.397083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.397004 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.497309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.497277 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:07:40.501484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.501464 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.516000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.515975 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:07:40.535051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.534944 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.541783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.541762 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.541910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.541811 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.541910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.541833 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.541910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.541904 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.542060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.542037 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.542123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.542105 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.542158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.542139 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fc5j\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.595970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.595937 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:07:40.595970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.595937 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:07:40.596154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.596003 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:07:40.596154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.596006 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:07:40.596154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.596042 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8tt9h\"" Oct 31 00:07:40.643224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643181 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643214 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643235 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643261 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643288 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643306 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fc5j\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.643438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.643353 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.644036 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.644005 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.646213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.646164 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.646334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.646287 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.646641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.646615 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.646806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.646787 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.647230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.647181 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.659904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.659872 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fc5j\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:40.662364 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.662336 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:40.824851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:40.824820 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:41.112537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:41.112496 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:07:41.115406 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:41.115380 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0697706_19ed_4104_a74a_031d7c04bb94.slice/crio-d090d6982e0ac464eb3a92b3c709276e74c87cc5bd89fbe4e5c5eceef845eea5 WatchSource:0}: Error finding container d090d6982e0ac464eb3a92b3c709276e74c87cc5bd89fbe4e5c5eceef845eea5: Status 404 returned error can't find the container with id d090d6982e0ac464eb3a92b3c709276e74c87cc5bd89fbe4e5c5eceef845eea5 Oct 31 00:07:42.043859 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.043824 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerStarted","Data":"d090d6982e0ac464eb3a92b3c709276e74c87cc5bd89fbe4e5c5eceef845eea5"} Oct 31 00:07:42.357262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.357178 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.372258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.367302 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.458507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.458475 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.458637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.458620 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.462119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.462091 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.555615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.555585 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.568869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.568846 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.694972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.694889 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:42.770512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.770484 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:42.907810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.907767 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:42.976638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:42.976569 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:07:42.980276 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:07:42.980250 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c534f0d_a863_43e0_8d5b_23dd34af02cb.slice/crio-4369f11e80ee4f203077f3926a5fd87e68a9373b7286b0cbf0fa2b265eb502ee WatchSource:0}: Error finding container 4369f11e80ee4f203077f3926a5fd87e68a9373b7286b0cbf0fa2b265eb502ee: Status 404 returned error can't find the container with id 4369f11e80ee4f203077f3926a5fd87e68a9373b7286b0cbf0fa2b265eb502ee Oct 31 00:07:43.048125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:43.048098 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"8c534f0d-a863-43e0-8d5b-23dd34af02cb","Type":"ContainerStarted","Data":"4369f11e80ee4f203077f3926a5fd87e68a9373b7286b0cbf0fa2b265eb502ee"} Oct 31 00:07:43.695237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:43.695150 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:44.055409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:44.055367 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"8c534f0d-a863-43e0-8d5b-23dd34af02cb","Type":"ContainerStarted","Data":"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b"} Oct 31 00:07:44.055572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:44.055426 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:44.057068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:44.057034 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerStarted","Data":"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb"} Oct 31 00:07:44.075572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:44.074450 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=11.07443591 podStartE2EDuration="11.07443591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:07:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:44.074070438 +0000 UTC m=+503.065730048" watchObservedRunningTime="2026-10-31 00:07:44.07443591 +0000 UTC m=+503.066095495" Oct 31 00:07:47.073108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:47.073076 5577 generic.go:356] "Generic (PLEG): container finished" podID="c0697706-19ed-4104-a74a-031d7c04bb94" containerID="68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb" exitCode=0 Oct 31 00:07:47.073108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:47.073111 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerDied","Data":"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb"} Oct 31 00:07:47.074558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:47.074538 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:07:50.088231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:50.088201 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerStarted","Data":"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5"} Oct 31 00:07:50.088528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:50.088273 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerStarted","Data":"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d"} Oct 31 00:07:50.088528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:50.088494 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:50.116609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:50.116563 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.048468051 podStartE2EDuration="10.116549118s" totalImagesPullingTime="4.472422089s" totalInitContainerRuntime="3.595658978s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:07:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:07:50.114842148 +0000 UTC m=+509.106501729" watchObservedRunningTime="2026-10-31 00:07:50.116549118 +0000 UTC m=+509.108208701" Oct 31 00:07:52.176410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:52.176376 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:52.308464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:52.308407 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:07:55.064940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:55.064914 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:07:56.094358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:07:56.094330 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:07:59.292695 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:07:59.292661 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:08:07.141983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.141954 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:08:07.142404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.142181 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="kube-rbac-proxy" containerID="cri-o://4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" gracePeriod=30 Oct 31 00:08:07.142404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.142261 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="prom-label-proxy" containerID="cri-o://4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" gracePeriod=30 Oct 31 00:08:07.142404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.142274 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="method-gate" containerID="cri-o://873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" gracePeriod=30 Oct 31 00:08:07.150309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.150271 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:08:07.150540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.150517 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="kube-rbac-proxy" containerID="cri-o://f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" gracePeriod=30 Oct 31 00:08:07.150620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.150577 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="prom-label-proxy" containerID="cri-o://eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" gracePeriod=30 Oct 31 00:08:07.150738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.150686 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="method-gate" containerID="cri-o://52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" gracePeriod=30 Oct 31 00:08:07.406242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.406091 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:07.412537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.412501 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:08:07.412823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.412791 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" containerID="cri-o://a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d" gracePeriod=120 Oct 31 00:08:07.413144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.413122 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="config-reloader" containerID="cri-o://ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5" gracePeriod=120 Oct 31 00:08:07.433927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.433902 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:08:07.437739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.437702 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:08:07.553194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.553162 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:07.570545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.570517 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:07.570753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.570703 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" containerName="otc-container" containerID="cri-o://421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79" gracePeriod=30 Oct 31 00:08:07.586861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.586837 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\") pod \"c842037b-4674-4374-8b9d-2f21d06b80d4\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " Oct 31 00:08:07.587007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.586901 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\" (UniqueName: \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\") pod \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " Oct 31 00:08:07.587007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.586938 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") pod \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " Oct 31 00:08:07.587007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.586964 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\" (UniqueName: \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\") pod \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\" (UID: \"66e9802d-8e7d-4a37-8352-a7f05b60ada6\") " Oct 31 00:08:07.587007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.586996 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\") pod \"c842037b-4674-4374-8b9d-2f21d06b80d4\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " Oct 31 00:08:07.587233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587025 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\") pod \"c842037b-4674-4374-8b9d-2f21d06b80d4\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " Oct 31 00:08:07.587233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587085 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\" (UniqueName: \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\") pod \"c842037b-4674-4374-8b9d-2f21d06b80d4\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " Oct 31 00:08:07.587233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587121 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\" (UniqueName: \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\") pod \"c842037b-4674-4374-8b9d-2f21d06b80d4\" (UID: \"c842037b-4674-4374-8b9d-2f21d06b80d4\") " Oct 31 00:08:07.587377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587232 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca" pod "c842037b-4674-4374-8b9d-2f21d06b80d4" (UID: "c842037b-4674-4374-8b9d-2f21d06b80d4"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.587377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587238 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config" pod "66e9802d-8e7d-4a37-8352-a7f05b60ada6" (UID: "66e9802d-8e7d-4a37-8352-a7f05b60ada6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.587525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587502 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.587589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587532 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66e9802d-8e7d-4a37-8352-a7f05b60ada6-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.587649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587588 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca" pod "c842037b-4674-4374-8b9d-2f21d06b80d4" (UID: "c842037b-4674-4374-8b9d-2f21d06b80d4"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.588069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.587922 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config" pod "c842037b-4674-4374-8b9d-2f21d06b80d4" (UID: "c842037b-4674-4374-8b9d-2f21d06b80d4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:07.592513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.592479 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls" pod "c842037b-4674-4374-8b9d-2f21d06b80d4" (UID: "c842037b-4674-4374-8b9d-2f21d06b80d4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.595358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.595331 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6" pod "c842037b-4674-4374-8b9d-2f21d06b80d4" (UID: "c842037b-4674-4374-8b9d-2f21d06b80d4"). InnerVolumeSpecName "kube-api-access-mfpw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.595605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.595585 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls" pod "66e9802d-8e7d-4a37-8352-a7f05b60ada6" (UID: "66e9802d-8e7d-4a37-8352-a7f05b60ada6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:07.595738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.595694 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc" pod "66e9802d-8e7d-4a37-8352-a7f05b60ada6" (UID: "66e9802d-8e7d-4a37-8352-a7f05b60ada6"). InnerVolumeSpecName "kube-api-access-42lcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:07.613746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.609652 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:07.613746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.609869 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" containerName="perses" containerID="cri-o://3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b" gracePeriod=30 Oct 31 00:08:07.688420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688391 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/66e9802d-8e7d-4a37-8352-a7f05b60ada6-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.688420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688421 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42lcc\" (UniqueName: \"kubernetes.io/projected/66e9802d-8e7d-4a37-8352-a7f05b60ada6-kube-api-access-42lcc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.688618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688436 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.688618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688448 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c842037b-4674-4374-8b9d-2f21d06b80d4-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.688618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688460 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfpw6\" (UniqueName: \"kubernetes.io/projected/c842037b-4674-4374-8b9d-2f21d06b80d4-kube-api-access-mfpw6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.688618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.688471 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c842037b-4674-4374-8b9d-2f21d06b80d4-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:07.705155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.705128 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:07.849310 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.849273 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:07.882688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.882663 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:07.888670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.888644 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:07.990439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990312 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\" (UniqueName: \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\") pod \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " Oct 31 00:08:07.990439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990350 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:07.990439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990380 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\" (UniqueName: \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990460 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990497 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990529 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990558 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990589 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\" (UniqueName: \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\") pod \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990640 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\" (UniqueName: \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\") pod \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\" (UID: \"8c534f0d-a863-43e0-8d5b-23dd34af02cb\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.990678 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\") pod \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\" (UID: \"d7d730e6-b4af-4498-a2ed-94e5346a27b1\") " Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.991310 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config" pod "8c534f0d-a863-43e0-8d5b-23dd34af02cb" (UID: "8c534f0d-a863-43e0-8d5b-23dd34af02cb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.991731 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.992992 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.993299 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.993382 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.993471 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m" pod "8c534f0d-a863-43e0-8d5b-23dd34af02cb" (UID: "8c534f0d-a863-43e0-8d5b-23dd34af02cb"). InnerVolumeSpecName "kube-api-access-pml9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.993520 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.994095 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "8c534f0d-a863-43e0-8d5b-23dd34af02cb" (UID: "8c534f0d-a863-43e0-8d5b-23dd34af02cb"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.994366 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb" pod "d7d730e6-b4af-4498-a2ed-94e5346a27b1" (UID: "d7d730e6-b4af-4498-a2ed-94e5346a27b1"). InnerVolumeSpecName "kube-api-access-zqvrb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:08.003401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:07.995249 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:08.091272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091238 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091263 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091274 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d7d730e6-b4af-4498-a2ed-94e5346a27b1-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091284 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8c534f0d-a863-43e0-8d5b-23dd34af02cb-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091293 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091304 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pml9m\" (UniqueName: \"kubernetes.io/projected/8c534f0d-a863-43e0-8d5b-23dd34af02cb-kube-api-access-pml9m\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091313 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d7d730e6-b4af-4498-a2ed-94e5346a27b1-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091321 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqvrb\" (UniqueName: \"kubernetes.io/projected/d7d730e6-b4af-4498-a2ed-94e5346a27b1-kube-api-access-zqvrb\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.091546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.091348 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:08:08.099865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.099845 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:08:08.136304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.136269 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins" pod "8c534f0d-a863-43e0-8d5b-23dd34af02cb" (UID: "8c534f0d-a863-43e0-8d5b-23dd34af02cb"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:08.167604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.167574 5577 generic.go:356] "Generic (PLEG): container finished" podID="c0697706-19ed-4104-a74a-031d7c04bb94" containerID="ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5" exitCode=0 Oct 31 00:08:08.168038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.167651 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerDied","Data":"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5"} Oct 31 00:08:08.169832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169811 5577 generic.go:356] "Generic (PLEG): container finished" podID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" exitCode=0 Oct 31 00:08:08.169832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169828 5577 generic.go:356] "Generic (PLEG): container finished" podID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" exitCode=0 Oct 31 00:08:08.169832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169836 5577 generic.go:356] "Generic (PLEG): container finished" podID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" exitCode=0 Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169861 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerDied","Data":"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633"} Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169879 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169927 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerDied","Data":"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd"} Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169969 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerDied","Data":"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e"} Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.170006 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn" event={"ID":"66e9802d-8e7d-4a37-8352-a7f05b60ada6","Type":"ContainerDied","Data":"da369674a4db7663b9333cefd7a1ad1664ac952818d55a9603648b98554863e2"} Oct 31 00:08:08.170059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.169988 5577 scope.go:122] "RemoveContainer" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" Oct 31 00:08:08.171307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.171190 5577 generic.go:356] "Generic (PLEG): container finished" podID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" containerID="3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b" exitCode=0 Oct 31 00:08:08.171307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.171213 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"8c534f0d-a863-43e0-8d5b-23dd34af02cb","Type":"ContainerDied","Data":"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b"} Oct 31 00:08:08.171307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.171246 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"8c534f0d-a863-43e0-8d5b-23dd34af02cb","Type":"ContainerDied","Data":"4369f11e80ee4f203077f3926a5fd87e68a9373b7286b0cbf0fa2b265eb502ee"} Oct 31 00:08:08.171307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.171252 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:08.177579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177556 5577 generic.go:356] "Generic (PLEG): container finished" podID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" exitCode=0 Oct 31 00:08:08.177579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177576 5577 generic.go:356] "Generic (PLEG): container finished" podID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" exitCode=0 Oct 31 00:08:08.177579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177582 5577 generic.go:356] "Generic (PLEG): container finished" podID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" exitCode=0 Oct 31 00:08:08.177838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177613 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" Oct 31 00:08:08.177838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177645 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerDied","Data":"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3"} Oct 31 00:08:08.177838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177721 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerDied","Data":"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70"} Oct 31 00:08:08.177838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177762 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerDied","Data":"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4"} Oct 31 00:08:08.177838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.177797 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth" event={"ID":"c842037b-4674-4374-8b9d-2f21d06b80d4","Type":"ContainerDied","Data":"281c9d8543cb79c914c7f9b984b7b157cd333a01e853cb2380ca2f5ea4e5784f"} Oct 31 00:08:08.184767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.180909 5577 scope.go:122] "RemoveContainer" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" Oct 31 00:08:08.184767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.180975 5577 generic.go:356] "Generic (PLEG): container finished" podID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" containerID="421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79" exitCode=0 Oct 31 00:08:08.184767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.181005 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d7d730e6-b4af-4498-a2ed-94e5346a27b1","Type":"ContainerDied","Data":"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79"} Oct 31 00:08:08.184767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.181017 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:08.184767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.181039 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d7d730e6-b4af-4498-a2ed-94e5346a27b1","Type":"ContainerDied","Data":"681dbbef670c5d8057f28531b33e0298b9e845450fa5ce64ad66641aab25062c"} Oct 31 00:08:08.191266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.191248 5577 scope.go:122] "RemoveContainer" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" Oct 31 00:08:08.191680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.191663 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:08:08.191680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.191681 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/8c534f0d-a863-43e0-8d5b-23dd34af02cb-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:08.199088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.199067 5577 scope.go:122] "RemoveContainer" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" Oct 31 00:08:08.199374 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.199351 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": container with ID starting with eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633 not found: ID does not exist" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" Oct 31 00:08:08.199466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.199384 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633"} err="failed to get container status \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": rpc error: code = NotFound desc = could not find container \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": container with ID starting with eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633 not found: ID does not exist" Oct 31 00:08:08.199466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.199405 5577 scope.go:122] "RemoveContainer" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" Oct 31 00:08:08.199864 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.199736 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": container with ID starting with 52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd not found: ID does not exist" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" Oct 31 00:08:08.199864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.199770 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd"} err="failed to get container status \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": rpc error: code = NotFound desc = could not find container \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": container with ID starting with 52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd not found: ID does not exist" Oct 31 00:08:08.199864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.199792 5577 scope.go:122] "RemoveContainer" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" Oct 31 00:08:08.200148 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.200130 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": container with ID starting with f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e not found: ID does not exist" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" Oct 31 00:08:08.200215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200152 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e"} err="failed to get container status \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": rpc error: code = NotFound desc = could not find container \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": container with ID starting with f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e not found: ID does not exist" Oct 31 00:08:08.200215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200167 5577 scope.go:122] "RemoveContainer" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" Oct 31 00:08:08.200471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200422 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633"} err="failed to get container status \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": rpc error: code = NotFound desc = could not find container \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": container with ID starting with eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633 not found: ID does not exist" Oct 31 00:08:08.200471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200445 5577 scope.go:122] "RemoveContainer" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" Oct 31 00:08:08.200599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200582 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:08:08.200749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200698 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd"} err="failed to get container status \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": rpc error: code = NotFound desc = could not find container \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": container with ID starting with 52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd not found: ID does not exist" Oct 31 00:08:08.200749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200748 5577 scope.go:122] "RemoveContainer" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" Oct 31 00:08:08.200959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200943 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e"} err="failed to get container status \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": rpc error: code = NotFound desc = could not find container \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": container with ID starting with f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e not found: ID does not exist" Oct 31 00:08:08.201002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.200959 5577 scope.go:122] "RemoveContainer" containerID="eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633" Oct 31 00:08:08.201147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201132 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633"} err="failed to get container status \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": rpc error: code = NotFound desc = could not find container \"eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633\": container with ID starting with eebd2d613d51d5baaad9a5fa274c23d1f706d6f99787d235909ce2922c780633 not found: ID does not exist" Oct 31 00:08:08.201189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201147 5577 scope.go:122] "RemoveContainer" containerID="52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd" Oct 31 00:08:08.201323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201308 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd"} err="failed to get container status \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": rpc error: code = NotFound desc = could not find container \"52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd\": container with ID starting with 52dec472c73ef2e56776e99b47cffd1a44f1726a5eea565f89591d49e851f5cd not found: ID does not exist" Oct 31 00:08:08.201369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201323 5577 scope.go:122] "RemoveContainer" containerID="f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e" Oct 31 00:08:08.201493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201472 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e"} err="failed to get container status \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": rpc error: code = NotFound desc = could not find container \"f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e\": container with ID starting with f0420473120ddc4dc703cc5d69d31715aa9ec7f42973671b39f2be6def5caa4e not found: ID does not exist" Oct 31 00:08:08.201493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.201492 5577 scope.go:122] "RemoveContainer" containerID="3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b" Oct 31 00:08:08.208047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.208024 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pqnnn"] Oct 31 00:08:08.208898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.208885 5577 scope.go:122] "RemoveContainer" containerID="3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b" Oct 31 00:08:08.209143 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.209128 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b\": container with ID starting with 3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b not found: ID does not exist" containerID="3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b" Oct 31 00:08:08.209200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.209147 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b"} err="failed to get container status \"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b\": rpc error: code = NotFound desc = could not find container \"3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b\": container with ID starting with 3d4914d14eca392d71ac57e2b8766c9f9e5152b7c210514872fd30a6d1138c8b not found: ID does not exist" Oct 31 00:08:08.209200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.209165 5577 scope.go:122] "RemoveContainer" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" Oct 31 00:08:08.216395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.216379 5577 scope.go:122] "RemoveContainer" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" Oct 31 00:08:08.220158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.220139 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:08:08.223763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.223747 5577 scope.go:122] "RemoveContainer" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" Oct 31 00:08:08.227157 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.227135 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-6bkth"] Oct 31 00:08:08.231844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.231829 5577 scope.go:122] "RemoveContainer" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" Oct 31 00:08:08.232096 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.232077 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": container with ID starting with 4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3 not found: ID does not exist" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" Oct 31 00:08:08.232149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232101 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3"} err="failed to get container status \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": rpc error: code = NotFound desc = could not find container \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": container with ID starting with 4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3 not found: ID does not exist" Oct 31 00:08:08.232149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232124 5577 scope.go:122] "RemoveContainer" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" Oct 31 00:08:08.232357 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.232340 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": container with ID starting with 873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70 not found: ID does not exist" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" Oct 31 00:08:08.232398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232364 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70"} err="failed to get container status \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": rpc error: code = NotFound desc = could not find container \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": container with ID starting with 873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70 not found: ID does not exist" Oct 31 00:08:08.232398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232377 5577 scope.go:122] "RemoveContainer" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" Oct 31 00:08:08.232611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.232594 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": container with ID starting with 4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4 not found: ID does not exist" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" Oct 31 00:08:08.232689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232616 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4"} err="failed to get container status \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": rpc error: code = NotFound desc = could not find container \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": container with ID starting with 4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4 not found: ID does not exist" Oct 31 00:08:08.232689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232628 5577 scope.go:122] "RemoveContainer" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" Oct 31 00:08:08.232855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232838 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3"} err="failed to get container status \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": rpc error: code = NotFound desc = could not find container \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": container with ID starting with 4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3 not found: ID does not exist" Oct 31 00:08:08.232896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.232855 5577 scope.go:122] "RemoveContainer" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" Oct 31 00:08:08.233031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233011 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70"} err="failed to get container status \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": rpc error: code = NotFound desc = could not find container \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": container with ID starting with 873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70 not found: ID does not exist" Oct 31 00:08:08.233071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233032 5577 scope.go:122] "RemoveContainer" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" Oct 31 00:08:08.233212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233192 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4"} err="failed to get container status \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": rpc error: code = NotFound desc = could not find container \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": container with ID starting with 4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4 not found: ID does not exist" Oct 31 00:08:08.233251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233214 5577 scope.go:122] "RemoveContainer" containerID="4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3" Oct 31 00:08:08.233382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233366 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3"} err="failed to get container status \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": rpc error: code = NotFound desc = could not find container \"4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3\": container with ID starting with 4d7ed5adac443e6da1c67e2f9c0770a9fd5e89407bad23f012261daecd5861f3 not found: ID does not exist" Oct 31 00:08:08.233424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233381 5577 scope.go:122] "RemoveContainer" containerID="873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70" Oct 31 00:08:08.233605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233587 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70"} err="failed to get container status \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": rpc error: code = NotFound desc = could not find container \"873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70\": container with ID starting with 873e22752d7f1e9872d774bd4e65011f337d0a266644e280912e6a2ba7a33e70 not found: ID does not exist" Oct 31 00:08:08.233605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233604 5577 scope.go:122] "RemoveContainer" containerID="4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4" Oct 31 00:08:08.233797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233782 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4"} err="failed to get container status \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": rpc error: code = NotFound desc = could not find container \"4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4\": container with ID starting with 4c128157ad7e8764813b70b71bb99dfa850d9713c252e50052bee1cffe1047a4 not found: ID does not exist" Oct 31 00:08:08.233845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.233797 5577 scope.go:122] "RemoveContainer" containerID="421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79" Oct 31 00:08:08.239255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.239237 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:08.240829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.240816 5577 scope.go:122] "RemoveContainer" containerID="421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79" Oct 31 00:08:08.241049 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:08.241032 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79\": container with ID starting with 421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79 not found: ID does not exist" containerID="421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79" Oct 31 00:08:08.241121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.241052 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79"} err="failed to get container status \"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79\": rpc error: code = NotFound desc = could not find container \"421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79\": container with ID starting with 421e724b525810068d7ca5f9a4a8e96d2f6413474cfaf583683dc675a928ab79 not found: ID does not exist" Oct 31 00:08:08.243794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.243775 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:08.255620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.255601 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:08.259361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:08.259342 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:09.129883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:09.129853 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" path="/var/lib/kubelet/pods/66e9802d-8e7d-4a37-8352-a7f05b60ada6/volumes" Oct 31 00:08:09.130313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:09.130301 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" path="/var/lib/kubelet/pods/8c534f0d-a863-43e0-8d5b-23dd34af02cb/volumes" Oct 31 00:08:09.130620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:09.130609 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" path="/var/lib/kubelet/pods/c842037b-4674-4374-8b9d-2f21d06b80d4/volumes" Oct 31 00:08:09.131326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:09.131313 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" path="/var/lib/kubelet/pods/d7d730e6-b4af-4498-a2ed-94e5346a27b1/volumes" Oct 31 00:08:10.047294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.047257 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:10.181365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.181332 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:10.249348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249318 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:10.249743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249698 5577 cpu_manager.go:414] "containerMap: removing container" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="method-gate" Oct 31 00:08:10.249743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249742 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="method-gate" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249758 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" containerName="otc-container" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249765 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" containerName="otc-container" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249772 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="prom-label-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249780 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="prom-label-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249789 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" containerName="perses" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249796 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" containerName="perses" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249834 5577 cpu_manager.go:414] "containerMap: removing container" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="prom-label-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249841 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="prom-label-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249853 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="kube-rbac-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249859 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="kube-rbac-proxy" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249870 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="method-gate" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249876 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="method-gate" Oct 31 00:08:10.249892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249891 5577 cpu_manager.go:414] "containerMap: removing container" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="kube-rbac-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249898 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="kube-rbac-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249976 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="prom-label-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249985 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="kube-rbac-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.249995 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="method-gate" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.250003 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="kube-rbac-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.250015 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="66e9802d-8e7d-4a37-8352-a7f05b60ada6" containerName="method-gate" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.250023 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d730e6-b4af-4498-a2ed-94e5346a27b1" containerName="otc-container" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.250030 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c842037b-4674-4374-8b9d-2f21d06b80d4" containerName="prom-label-proxy" Oct 31 00:08:10.250274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.250036 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c534f0d-a863-43e0-8d5b-23dd34af02cb" containerName="perses" Oct 31 00:08:10.267428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.267400 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.270774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.270750 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:10.273255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.273235 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:08:10.275545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.275520 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:08:10.276702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.276683 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:08:10.288283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.288254 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:10.291125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.291100 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.301029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.301005 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:10.324264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.324242 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:10.373798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.373774 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:08:10.378121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.378105 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:08:10.411470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411445 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr8h9\" (UniqueName: \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.411561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411474 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.411561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411500 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dpzq\" (UniqueName: \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.411561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411549 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.411658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411622 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.411658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411638 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.411736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.411665 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.512185 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512162 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.512299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512191 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.512299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512226 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.512299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512274 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr8h9\" (UniqueName: \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.512445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512316 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.512445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512351 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dpzq\" (UniqueName: \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.512445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512373 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.512704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.512684 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.513074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.513054 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.513143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.513055 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.514890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.514864 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.514996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.514980 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.524744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.524727 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr8h9\" (UniqueName: \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:10.524828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.524745 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dpzq\" (UniqueName: \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\") pod \"data-science-collector-collector-0\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.551217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.551138 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:10.582174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.582146 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:08:10.679587 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.679548 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:10.859285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.859214 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:08:10.862175 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:08:10.862138 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8ca0c6_983c_4cea_bc90_2e44fe4ec6d4.slice/crio-52245f2f19801c419759a7379923f08cf05b61566ae33d421b3f968d5f414d4b WatchSource:0}: Error finding container 52245f2f19801c419759a7379923f08cf05b61566ae33d421b3f968d5f414d4b: Status 404 returned error can't find the container with id 52245f2f19801c419759a7379923f08cf05b61566ae33d421b3f968d5f414d4b Oct 31 00:08:10.864286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.864253 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:08:10.864393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:10.864316 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:08:11.080440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.080394 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:11.093635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.093610 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.31:9093/-/ready\": dial tcp 10.133.0.31:9093: connect: connection refused" Oct 31 00:08:11.201336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.201252 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4","Type":"ContainerStarted","Data":"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509"} Oct 31 00:08:11.201336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.201299 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4","Type":"ContainerStarted","Data":"52245f2f19801c419759a7379923f08cf05b61566ae33d421b3f968d5f414d4b"} Oct 31 00:08:11.222019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.221969 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.221954771 podStartE2EDuration="1.221954771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:11.221052622 +0000 UTC m=+530.212712210" watchObservedRunningTime="2026-10-31 00:08:11.221954771 +0000 UTC m=+530.213614349" Oct 31 00:08:11.261470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:11.261437 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:13.133940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.133911 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.141996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.141976 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.235182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.235151 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.235363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.235321 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.238644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.238625 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.326703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.326674 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.339671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.339646 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.605831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.605797 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:13.876356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:13.876276 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:08:13.880240 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:08:13.880210 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c330ba7_f586_4e8f_ab2c_4efe1c49db98.slice/crio-59761ffbbf0d871bf20f4b04ccda607e985c29a78430fd2208acdf14177992cf WatchSource:0}: Error finding container 59761ffbbf0d871bf20f4b04ccda607e985c29a78430fd2208acdf14177992cf: Status 404 returned error can't find the container with id 59761ffbbf0d871bf20f4b04ccda607e985c29a78430fd2208acdf14177992cf Oct 31 00:08:14.215538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:14.215456 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0c330ba7-f586-4e8f-ab2c-4efe1c49db98","Type":"ContainerStarted","Data":"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f"} Oct 31 00:08:14.215538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:14.215505 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0c330ba7-f586-4e8f-ab2c-4efe1c49db98","Type":"ContainerStarted","Data":"59761ffbbf0d871bf20f4b04ccda607e985c29a78430fd2208acdf14177992cf"} Oct 31 00:08:14.216085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:14.215547 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:14.235104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:14.235051 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.23503915 podStartE2EDuration="4.23503915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:08:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:08:14.234771154 +0000 UTC m=+533.226430739" watchObservedRunningTime="2026-10-31 00:08:14.23503915 +0000 UTC m=+533.226698728" Oct 31 00:08:16.093957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:16.093918 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.31:9093/-/ready\": dial tcp 10.133.0.31:9093: connect: connection refused" Oct 31 00:08:17.655534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.655512 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:08:17.777133 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777100 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777156 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777186 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777231 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777265 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777295 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777345 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\") pod \"c0697706-19ed-4104-a74a-031d7c04bb94\" (UID: \"c0697706-19ed-4104-a74a-031d7c04bb94\") " Oct 31 00:08:17.777563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.777519 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:17.779805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.779772 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "kube-api-access-5fc5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:17.779949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.779919 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:17.780067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.780043 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:08:17.780103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.780050 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:08:17.783778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.783735 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:17.791645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.791618 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config" pod "c0697706-19ed-4104-a74a-031d7c04bb94" (UID: "c0697706-19ed-4104-a74a-031d7c04bb94"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:08:17.877918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877870 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.877918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877903 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.877918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877912 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.877918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877922 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fc5j\" (UniqueName: \"kubernetes.io/projected/c0697706-19ed-4104-a74a-031d7c04bb94-kube-api-access-5fc5j\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.878255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877934 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.878255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877943 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c0697706-19ed-4104-a74a-031d7c04bb94-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:17.878255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:17.877951 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c0697706-19ed-4104-a74a-031d7c04bb94-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:08:18.234869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.234788 5577 generic.go:356] "Generic (PLEG): container finished" podID="c0697706-19ed-4104-a74a-031d7c04bb94" containerID="a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d" exitCode=0 Oct 31 00:08:18.234869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.234825 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerDied","Data":"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d"} Oct 31 00:08:18.234869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.234858 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:08:18.235128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.234873 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"c0697706-19ed-4104-a74a-031d7c04bb94","Type":"ContainerDied","Data":"d090d6982e0ac464eb3a92b3c709276e74c87cc5bd89fbe4e5c5eceef845eea5"} Oct 31 00:08:18.235128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.234901 5577 scope.go:122] "RemoveContainer" containerID="ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5" Oct 31 00:08:18.244067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.244049 5577 scope.go:122] "RemoveContainer" containerID="a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d" Oct 31 00:08:18.251988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.251970 5577 scope.go:122] "RemoveContainer" containerID="68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb" Oct 31 00:08:18.259495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.259477 5577 scope.go:122] "RemoveContainer" containerID="ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5" Oct 31 00:08:18.259842 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:18.259812 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5\": container with ID starting with ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5 not found: ID does not exist" containerID="ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5" Oct 31 00:08:18.260008 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.259853 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5"} err="failed to get container status \"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5\": rpc error: code = NotFound desc = could not find container \"ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5\": container with ID starting with ec8f11a01d2ee9032bdb40c6b8491e64e0a9d887cd2f093808242b73839fa3a5 not found: ID does not exist" Oct 31 00:08:18.260008 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.259877 5577 scope.go:122] "RemoveContainer" containerID="a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d" Oct 31 00:08:18.260408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.260260 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:08:18.260408 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:18.260269 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d\": container with ID starting with a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d not found: ID does not exist" containerID="a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d" Oct 31 00:08:18.260408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.260321 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d"} err="failed to get container status \"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d\": rpc error: code = NotFound desc = could not find container \"a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d\": container with ID starting with a7b355dc9f6394d8059263cf3f7801eb6dc8fb5774de3703571a8ae08570aa2d not found: ID does not exist" Oct 31 00:08:18.260408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.260341 5577 scope.go:122] "RemoveContainer" containerID="68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb" Oct 31 00:08:18.260656 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:18.260636 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb\": container with ID starting with 68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb not found: ID does not exist" containerID="68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb" Oct 31 00:08:18.260693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.260666 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb"} err="failed to get container status \"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb\": rpc error: code = NotFound desc = could not find container \"68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb\": container with ID starting with 68abe9f41a73e450d6a5a4d77fbbf86ce41a6aea6abfe186657357e8178e3bdb not found: ID does not exist" Oct 31 00:08:18.264355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:18.264334 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:08:19.129620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:19.129586 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" path="/var/lib/kubelet/pods/c0697706-19ed-4104-a74a-031d7c04bb94/volumes" Oct 31 00:08:21.491650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:21.491531 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:21.624352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:21.624310 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:24.994072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:24.994034 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:25.221523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:25.221493 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:08:29.292577 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:29.292492 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:08:51.708068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:08:51.708029 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:08:59.291621 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:08:59.291581 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:09:00.307317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.307287 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:00.307738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.307492 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" containerName="otc-container" containerID="cri-o://c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509" gracePeriod=30 Oct 31 00:09:00.348547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.348518 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:00.348781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.348741 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" containerName="perses" containerID="cri-o://3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f" gracePeriod=30 Oct 31 00:09:00.398695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.398662 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:00.527782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.527751 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:00.587520 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.587498 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:00.620456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.620429 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:00.705506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705466 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\") pod \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " Oct 31 00:09:00.705506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705517 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\" (UniqueName: \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\") pod \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " Oct 31 00:09:00.705761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705539 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\" (UniqueName: \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\") pod \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " Oct 31 00:09:00.705761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705560 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\") pod \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " Oct 31 00:09:00.705761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705663 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " Oct 31 00:09:00.705761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705693 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\" (UniqueName: \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\") pod \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " Oct 31 00:09:00.705963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705769 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\") pod \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\" (UID: \"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4\") " Oct 31 00:09:00.705963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705802 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\" (UniqueName: \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\") pod \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\" (UID: \"0c330ba7-f586-4e8f-ab2c-4efe1c49db98\") " Oct 31 00:09:00.705963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.705916 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal" pod "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" (UID: "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:00.706401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.706264 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config" pod "0c330ba7-f586-4e8f-ab2c-4efe1c49db98" (UID: "0c330ba7-f586-4e8f-ab2c-4efe1c49db98"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:00.708647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.708447 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs" pod "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" (UID: "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:00.708647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.708639 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9" pod "0c330ba7-f586-4e8f-ab2c-4efe1c49db98" (UID: "0c330ba7-f586-4e8f-ab2c-4efe1c49db98"). InnerVolumeSpecName "kube-api-access-kr8h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:00.708815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.708644 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs" pod "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" (UID: "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:09:00.708815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.708665 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq" pod "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" (UID: "ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4"). InnerVolumeSpecName "kube-api-access-9dpzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:00.708815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.708734 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "0c330ba7-f586-4e8f-ab2c-4efe1c49db98" (UID: "0c330ba7-f586-4e8f-ab2c-4efe1c49db98"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:09:00.806902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806863 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr8h9\" (UniqueName: \"kubernetes.io/projected/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-kube-api-access-kr8h9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.806902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806895 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dpzq\" (UniqueName: \"kubernetes.io/projected/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-kube-api-access-9dpzq\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.806902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806909 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.807135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806944 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:09:00.807135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806955 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.807135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806965 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.807135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.806981 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:00.821306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.821250 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:09:00.852759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.852732 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins" pod "0c330ba7-f586-4e8f-ab2c-4efe1c49db98" (UID: "0c330ba7-f586-4e8f-ab2c-4efe1c49db98"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:09:00.908200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.908176 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:09:00.908200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:00.908197 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0c330ba7-f586-4e8f-ab2c-4efe1c49db98-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:01.414152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.414065 5577 generic.go:356] "Generic (PLEG): container finished" podID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" containerID="c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509" exitCode=0 Oct 31 00:09:01.414152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.414124 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:01.414636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.414153 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4","Type":"ContainerDied","Data":"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509"} Oct 31 00:09:01.414636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.414198 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4","Type":"ContainerDied","Data":"52245f2f19801c419759a7379923f08cf05b61566ae33d421b3f968d5f414d4b"} Oct 31 00:09:01.414636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.414227 5577 scope.go:122] "RemoveContainer" containerID="c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509" Oct 31 00:09:01.415649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.415489 5577 generic.go:356] "Generic (PLEG): container finished" podID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" containerID="3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f" exitCode=0 Oct 31 00:09:01.415649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.415546 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:01.415649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.415541 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0c330ba7-f586-4e8f-ab2c-4efe1c49db98","Type":"ContainerDied","Data":"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f"} Oct 31 00:09:01.415649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.415595 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0c330ba7-f586-4e8f-ab2c-4efe1c49db98","Type":"ContainerDied","Data":"59761ffbbf0d871bf20f4b04ccda607e985c29a78430fd2208acdf14177992cf"} Oct 31 00:09:01.422489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.422473 5577 scope.go:122] "RemoveContainer" containerID="c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509" Oct 31 00:09:01.422780 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:01.422757 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509\": container with ID starting with c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509 not found: ID does not exist" containerID="c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509" Oct 31 00:09:01.422832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.422791 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509"} err="failed to get container status \"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509\": rpc error: code = NotFound desc = could not find container \"c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509\": container with ID starting with c4370029f7152eca817e9a46a8bdc117b54cb137caf4406952a2ac7a23249509 not found: ID does not exist" Oct 31 00:09:01.422832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.422813 5577 scope.go:122] "RemoveContainer" containerID="3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f" Oct 31 00:09:01.429903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.429882 5577 scope.go:122] "RemoveContainer" containerID="3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f" Oct 31 00:09:01.430245 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:01.430187 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f\": container with ID starting with 3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f not found: ID does not exist" containerID="3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f" Oct 31 00:09:01.430245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.430221 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f"} err="failed to get container status \"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f\": rpc error: code = NotFound desc = could not find container \"3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f\": container with ID starting with 3860d6e0204b830e7c99dcf38d5ee53808ed518589af4b129a3e2368791fc34f not found: ID does not exist" Oct 31 00:09:01.430753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.430737 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:01.434864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.434846 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:01.446526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.446502 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:01.450608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:01.450592 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:03.129265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:03.129232 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" path="/var/lib/kubelet/pods/0c330ba7-f586-4e8f-ab2c-4efe1c49db98/volumes" Oct 31 00:09:03.129644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:03.129578 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" path="/var/lib/kubelet/pods/ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4/volumes" Oct 31 00:09:08.474226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.474190 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:08.567663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.567631 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:09:08.568003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.567987 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="init-config-reloader" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568004 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="init-config-reloader" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568014 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" containerName="perses" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568019 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" containerName="perses" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568040 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568045 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568052 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="config-reloader" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568057 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="config-reloader" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568067 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" containerName="otc-container" Oct 31 00:09:08.568074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568071 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" containerName="otc-container" Oct 31 00:09:08.568328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568115 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c330ba7-f586-4e8f-ab2c-4efe1c49db98" containerName="perses" Oct 31 00:09:08.568328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568124 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="alertmanager" Oct 31 00:09:08.568328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568130 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0697706-19ed-4104-a74a-031d7c04bb94" containerName="config-reloader" Oct 31 00:09:08.568328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.568136 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8ca0c6-983c-4cea-bc90-2e44fe4ec6d4" containerName="otc-container" Oct 31 00:09:08.591676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.591635 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:09:08.591838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.591704 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:09:08.591838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.591777 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.595593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.595571 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.602689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.602665 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:09:08.605039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.604946 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:09:08.605326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.605294 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-7ghjs\"" Oct 31 00:09:08.605326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.605304 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:09:08.605503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.605373 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-lc474\"" Oct 31 00:09:08.668696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668650 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668741 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l24ft\" (UniqueName: \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668760 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668782 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668806 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668805 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668826 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668848 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:09:08.668911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668852 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:09:08.669309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668931 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.669309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.668965 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.669309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.669031 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.669309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.669061 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.669309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.669105 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.769933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.769896 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.770101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.769958 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.770101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770007 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.770101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770028 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.770101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770044 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.770101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770065 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.770450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770105 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l24ft\" (UniqueName: \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.770450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770140 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.770891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.770866 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.771331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.771118 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.771331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.771155 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.771331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.771205 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.773771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.773440 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.774242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.774195 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.788095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.788053 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xvzmw\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.791077 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.791048 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l24ft\" (UniqueName: \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pjjp5\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:08.914470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.914434 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:09:08.920803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:08.920775 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:09:09.140876 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.140796 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:09.146273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.146234 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.154877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.154849 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:09:09.155031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.154903 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:09:09.155031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.154929 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:09:09.155737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.155659 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:09:09.155980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.155957 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:09:09.159209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.159184 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:09.173913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.173885 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:09.179840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.179679 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.189888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.189861 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:09.209836 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.209610 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:09:09.210237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.210210 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:09:09.211559 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:09:09.211527 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c07ac39_80a3_47b5_a338_0b8617fb4a39.slice/crio-4e2571d22f233a1269f0c83fe43a745e4ccab946449060e01a6d68bd5dcdfbf9 WatchSource:0}: Error finding container 4e2571d22f233a1269f0c83fe43a745e4ccab946449060e01a6d68bd5dcdfbf9: Status 404 returned error can't find the container with id 4e2571d22f233a1269f0c83fe43a745e4ccab946449060e01a6d68bd5dcdfbf9 Oct 31 00:09:09.213660 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:09:09.213620 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9561c21d_ebef_4f55_9309_01a542e3ad3b.slice/crio-cca369cd2f40a4a27b036b4cbb823f73a5d9b72ecd2bc2177c4c279e098bfee2 WatchSource:0}: Error finding container cca369cd2f40a4a27b036b4cbb823f73a5d9b72ecd2bc2177c4c279e098bfee2: Status 404 returned error can't find the container with id cca369cd2f40a4a27b036b4cbb823f73a5d9b72ecd2bc2177c4c279e098bfee2 Oct 31 00:09:09.254781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.254760 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:09:09.254927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.254909 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:09:09.275156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275134 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275175 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275248 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.275374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275287 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92n98\" (UniqueName: \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.275421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275402 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275424 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275444 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sk24\" (UniqueName: \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275520 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.275570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.275539 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.376190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376167 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376645 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376703 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376757 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376803 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92n98\" (UniqueName: \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376864 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376893 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.376930 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sk24\" (UniqueName: \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.377051 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.377510 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.377906 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:09.377971 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:09.378043 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs podName:393b10c8-c521-4b63-9e0d-a9e73853b7a9 nodeName:}" failed. No retries permitted until 2026-10-31 00:09:09.878020495 +0000 UTC m=+588.869680058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:09:09.380895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.378534 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.390539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.390514 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.390628 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.390601 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92n98\" (UniqueName: \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:09.391467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.391408 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.392545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.392524 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sk24\" (UniqueName: \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.393219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.393199 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.451387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.451348 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerStarted","Data":"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264"} Oct 31 00:09:09.451496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.451470 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerStarted","Data":"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf"} Oct 31 00:09:09.451556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.451515 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerStarted","Data":"4e2571d22f233a1269f0c83fe43a745e4ccab946449060e01a6d68bd5dcdfbf9"} Oct 31 00:09:09.453114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.453087 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerStarted","Data":"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d"} Oct 31 00:09:09.453190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.453139 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerStarted","Data":"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651"} Oct 31 00:09:09.453190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.453161 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerStarted","Data":"cca369cd2f40a4a27b036b4cbb823f73a5d9b72ecd2bc2177c4c279e098bfee2"} Oct 31 00:09:09.878726 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.878674 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:09.879668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.879643 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:09.882428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:09.882410 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:10.007567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.007521 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:10.062990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.062964 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:09:10.170576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.170489 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:10.305290 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.305262 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:10.343024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.343001 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:09:10.345517 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:09:10.345492 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b10c8_c521_4b63_9e0d_a9e73853b7a9.slice/crio-884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4 WatchSource:0}: Error finding container 884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4: Status 404 returned error can't find the container with id 884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4 Oct 31 00:09:10.347968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.347941 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:09:10.348039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.347995 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:09:10.461773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.461673 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerStarted","Data":"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b"} Oct 31 00:09:10.463791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.463767 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerStarted","Data":"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6"} Oct 31 00:09:10.465616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.465589 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"393b10c8-c521-4b63-9e0d-a9e73853b7a9","Type":"ContainerStarted","Data":"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a"} Oct 31 00:09:10.465749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.465641 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"393b10c8-c521-4b63-9e0d-a9e73853b7a9","Type":"ContainerStarted","Data":"884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4"} Oct 31 00:09:10.482411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.482364 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" podStartSLOduration=2.482350148 podStartE2EDuration="2.482350148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.481315701 +0000 UTC m=+589.472975335" watchObservedRunningTime="2026-10-31 00:09:10.482350148 +0000 UTC m=+589.474009732" Oct 31 00:09:10.506744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.506681 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" podStartSLOduration=2.5066679990000003 podStartE2EDuration="2.506667999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.506358137 +0000 UTC m=+589.498017724" watchObservedRunningTime="2026-10-31 00:09:10.506667999 +0000 UTC m=+589.498327576" Oct 31 00:09:10.528741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.528612 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.528596793 podStartE2EDuration="1.528596793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:10.528320266 +0000 UTC m=+589.519979853" watchObservedRunningTime="2026-10-31 00:09:10.528596793 +0000 UTC m=+589.520256381" Oct 31 00:09:10.556019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:10.555984 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:14.420339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.420308 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.426533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.426499 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.485763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.485725 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:14.521418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.521392 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.521618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.521492 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.524703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.524685 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.615558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.615528 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.634304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.634127 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:14.682431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.682354 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:14.816159 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.816121 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:14.896020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:14.895989 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:15.067097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.067066 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:09:15.072537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.072516 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.086593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.086560 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:09:15.103171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.103125 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:15.125483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125437 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125493 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125525 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125567 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125612 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125654 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w7ft\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.125917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.125743 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.157745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.157718 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:09:15.157745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.157740 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:09:15.157958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.157739 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:09:15.157958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.157818 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:09:15.157958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.157834 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-wvv2m\"" Oct 31 00:09:15.169722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.169686 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:09:15.172512 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:09:15.172488 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18e8ec3_6014_4f1d_b259_0b7737addcc6.slice/crio-52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034 WatchSource:0}: Error finding container 52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034: Status 404 returned error can't find the container with id 52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034 Oct 31 00:09:15.226517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226494 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226546 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w7ft\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226611 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226689 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226737 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226763 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.226861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.226807 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.227114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.227073 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.229323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.229301 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.229513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.229484 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.229556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.229489 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.229556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.229533 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.229786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.229771 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.231987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.231963 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:15.238728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.238685 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w7ft\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.392934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.392903 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:15.494727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.491859 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c18e8ec3-6014-4f1d-b259-0b7737addcc6","Type":"ContainerStarted","Data":"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d"} Oct 31 00:09:15.494727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.491924 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c18e8ec3-6014-4f1d-b259-0b7737addcc6","Type":"ContainerStarted","Data":"52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034"} Oct 31 00:09:15.494727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.492596 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:15.528136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.528086 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.52807149 podStartE2EDuration="6.52807149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:15.527785249 +0000 UTC m=+594.519444834" watchObservedRunningTime="2026-10-31 00:09:15.52807149 +0000 UTC m=+594.519731080" Oct 31 00:09:15.681847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:15.681809 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:09:15.685808 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:09:15.685774 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a27225_917d_456f_8c9b_92e0d6264853.slice/crio-7ffd6b5886840b43a5015767170e469f3bc0eb49171c8a8e183caa893dc3d9b2 WatchSource:0}: Error finding container 7ffd6b5886840b43a5015767170e469f3bc0eb49171c8a8e183caa893dc3d9b2: Status 404 returned error can't find the container with id 7ffd6b5886840b43a5015767170e469f3bc0eb49171c8a8e183caa893dc3d9b2 Oct 31 00:09:16.499581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:16.499548 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerStarted","Data":"5de30251cf85f50973a1a9c4498f30d121bd9781904faa803cfc0a0fab5df2a3"} Oct 31 00:09:16.500032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:16.499624 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerStarted","Data":"7ffd6b5886840b43a5015767170e469f3bc0eb49171c8a8e183caa893dc3d9b2"} Oct 31 00:09:19.519782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:19.519746 5577 generic.go:356] "Generic (PLEG): container finished" podID="28a27225-917d-456f-8c9b-92e0d6264853" containerID="5de30251cf85f50973a1a9c4498f30d121bd9781904faa803cfc0a0fab5df2a3" exitCode=0 Oct 31 00:09:19.520164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:19.519816 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerDied","Data":"5de30251cf85f50973a1a9c4498f30d121bd9781904faa803cfc0a0fab5df2a3"} Oct 31 00:09:20.526627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:20.526598 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerStarted","Data":"1678b244b09ded185925d459a181160ee24991d16e7c7708a47255adc648b0ec"} Oct 31 00:09:20.527013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:20.526656 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerStarted","Data":"b45f650259e79e0e7b430bd610c72645d28daaddfa70a65ccc81a48973d39955"} Oct 31 00:09:20.527013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:20.526862 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:20.555287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:20.555223 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.280352033 podStartE2EDuration="5.555203343s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.27485131s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:20.553802757 +0000 UTC m=+599.545462341" watchObservedRunningTime="2026-10-31 00:09:20.555203343 +0000 UTC m=+599.546862936" Oct 31 00:09:21.182650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:21.182620 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:09:21.182840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:21.182620 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:09:23.344485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:23.344445 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:25.044800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:25.044766 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:25.173205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:25.173175 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:09:26.532026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:26.532001 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:09:27.514235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:09:27.514206 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:09:29.293095 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:29.293051 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:09:59.292280 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:09:59.292202 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:10:01.358845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.358809 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:10:01.359316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.359073 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="kube-rbac-proxy" containerID="cri-o://9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" gracePeriod=30 Oct 31 00:10:01.359316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.359117 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="method-gate" containerID="cri-o://53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" gracePeriod=30 Oct 31 00:10:01.359316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.359112 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="prom-label-proxy" containerID="cri-o://17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" gracePeriod=30 Oct 31 00:10:01.362451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.362118 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:10:01.363026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.362993 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="kube-rbac-proxy" containerID="cri-o://fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" gracePeriod=30 Oct 31 00:10:01.363151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.363037 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="method-gate" containerID="cri-o://757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" gracePeriod=30 Oct 31 00:10:01.363151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.363010 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="prom-label-proxy" containerID="cri-o://c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" gracePeriod=30 Oct 31 00:10:01.570755 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.569657 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:01.574329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.574302 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:01.574526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.574507 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" containerID="cri-o://b45f650259e79e0e7b430bd610c72645d28daaddfa70a65ccc81a48973d39955" gracePeriod=120 Oct 31 00:10:01.575290 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.574798 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="config-reloader" containerID="cri-o://1678b244b09ded185925d459a181160ee24991d16e7c7708a47255adc648b0ec" gracePeriod=120 Oct 31 00:10:01.669798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.669779 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:10:01.674431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.674402 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:10:01.704727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.704684 5577 generic.go:356] "Generic (PLEG): container finished" podID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" exitCode=0 Oct 31 00:10:01.704727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.704729 5577 generic.go:356] "Generic (PLEG): container finished" podID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" exitCode=0 Oct 31 00:10:01.705010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.704988 5577 generic.go:356] "Generic (PLEG): container finished" podID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" exitCode=0 Oct 31 00:10:01.705104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.704918 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerDied","Data":"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6"} Oct 31 00:10:01.705104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.705068 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" Oct 31 00:10:01.705319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.705133 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerDied","Data":"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d"} Oct 31 00:10:01.705319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.705181 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerDied","Data":"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651"} Oct 31 00:10:01.705319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.705219 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw" event={"ID":"9561c21d-ebef-4f55-9309-01a542e3ad3b","Type":"ContainerDied","Data":"cca369cd2f40a4a27b036b4cbb823f73a5d9b72ecd2bc2177c4c279e098bfee2"} Oct 31 00:10:01.705319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.705259 5577 scope.go:122] "RemoveContainer" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" Oct 31 00:10:01.711115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.711087 5577 generic.go:356] "Generic (PLEG): container finished" podID="28a27225-917d-456f-8c9b-92e0d6264853" containerID="1678b244b09ded185925d459a181160ee24991d16e7c7708a47255adc648b0ec" exitCode=0 Oct 31 00:10:01.711240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.711214 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerDied","Data":"1678b244b09ded185925d459a181160ee24991d16e7c7708a47255adc648b0ec"} Oct 31 00:10:01.713808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.713767 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:01.714335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714311 5577 generic.go:356] "Generic (PLEG): container finished" podID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" exitCode=0 Oct 31 00:10:01.714335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714332 5577 generic.go:356] "Generic (PLEG): container finished" podID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" exitCode=0 Oct 31 00:10:01.714507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714341 5577 generic.go:356] "Generic (PLEG): container finished" podID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" exitCode=0 Oct 31 00:10:01.714507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714381 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" Oct 31 00:10:01.714507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714385 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerDied","Data":"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b"} Oct 31 00:10:01.714507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714459 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerDied","Data":"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264"} Oct 31 00:10:01.714741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714530 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerDied","Data":"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf"} Oct 31 00:10:01.714741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.714657 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5" event={"ID":"5c07ac39-80a3-47b5-a338-0b8617fb4a39","Type":"ContainerDied","Data":"4e2571d22f233a1269f0c83fe43a745e4ccab946449060e01a6d68bd5dcdfbf9"} Oct 31 00:10:01.718958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.718924 5577 scope.go:122] "RemoveContainer" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" Oct 31 00:10:01.731068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.731046 5577 scope.go:122] "RemoveContainer" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" Oct 31 00:10:01.740910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.740539 5577 scope.go:122] "RemoveContainer" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" Oct 31 00:10:01.741136 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.740916 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": container with ID starting with 17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6 not found: ID does not exist" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" Oct 31 00:10:01.741199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741153 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6"} err="failed to get container status \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": rpc error: code = NotFound desc = could not find container \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": container with ID starting with 17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6 not found: ID does not exist" Oct 31 00:10:01.741199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741178 5577 scope.go:122] "RemoveContainer" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" Oct 31 00:10:01.741499 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.741463 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": container with ID starting with 53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d not found: ID does not exist" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" Oct 31 00:10:01.741614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741503 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d"} err="failed to get container status \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": rpc error: code = NotFound desc = could not find container \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": container with ID starting with 53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d not found: ID does not exist" Oct 31 00:10:01.741614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741518 5577 scope.go:122] "RemoveContainer" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" Oct 31 00:10:01.742194 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.741904 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": container with ID starting with 9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651 not found: ID does not exist" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" Oct 31 00:10:01.742194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741928 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651"} err="failed to get container status \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": rpc error: code = NotFound desc = could not find container \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": container with ID starting with 9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651 not found: ID does not exist" Oct 31 00:10:01.742194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.741953 5577 scope.go:122] "RemoveContainer" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" Oct 31 00:10:01.742380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742224 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6"} err="failed to get container status \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": rpc error: code = NotFound desc = could not find container \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": container with ID starting with 17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6 not found: ID does not exist" Oct 31 00:10:01.742380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742236 5577 scope.go:122] "RemoveContainer" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" Oct 31 00:10:01.742486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742467 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d"} err="failed to get container status \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": rpc error: code = NotFound desc = could not find container \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": container with ID starting with 53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d not found: ID does not exist" Oct 31 00:10:01.742486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742479 5577 scope.go:122] "RemoveContainer" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" Oct 31 00:10:01.742759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742740 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651"} err="failed to get container status \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": rpc error: code = NotFound desc = could not find container \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": container with ID starting with 9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651 not found: ID does not exist" Oct 31 00:10:01.742759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742758 5577 scope.go:122] "RemoveContainer" containerID="17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6" Oct 31 00:10:01.743008 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.742990 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6"} err="failed to get container status \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": rpc error: code = NotFound desc = could not find container \"17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6\": container with ID starting with 17ac0c12c69a38c95fe5730e5507221779bc3681593b714724b9cda5b59178e6 not found: ID does not exist" Oct 31 00:10:01.743075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.743009 5577 scope.go:122] "RemoveContainer" containerID="53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d" Oct 31 00:10:01.743341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.743322 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d"} err="failed to get container status \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": rpc error: code = NotFound desc = could not find container \"53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d\": container with ID starting with 53a7e8a1c02c024b4197c1ad1610d0ca63fc2d8cb79bb477db8a60e441d8ed8d not found: ID does not exist" Oct 31 00:10:01.743341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.743341 5577 scope.go:122] "RemoveContainer" containerID="9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651" Oct 31 00:10:01.743629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.743599 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651"} err="failed to get container status \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": rpc error: code = NotFound desc = could not find container \"9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651\": container with ID starting with 9cbc4ff6730e543245b2843c9f547a0d068a9dc6db88055d3a887dc7e0c6d651 not found: ID does not exist" Oct 31 00:10:01.743629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.743628 5577 scope.go:122] "RemoveContainer" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" Oct 31 00:10:01.752105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.752083 5577 scope.go:122] "RemoveContainer" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" Oct 31 00:10:01.763746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.763057 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:01.763746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.763285 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" containerName="otc-container" containerID="cri-o://9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a" gracePeriod=30 Oct 31 00:10:01.779657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.779634 5577 scope.go:122] "RemoveContainer" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" Oct 31 00:10:01.796244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.795924 5577 scope.go:122] "RemoveContainer" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" Oct 31 00:10:01.796858 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.796816 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": container with ID starting with c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b not found: ID does not exist" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" Oct 31 00:10:01.796971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.796855 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b"} err="failed to get container status \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": rpc error: code = NotFound desc = could not find container \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": container with ID starting with c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b not found: ID does not exist" Oct 31 00:10:01.796971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.796880 5577 scope.go:122] "RemoveContainer" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" Oct 31 00:10:01.797478 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.797360 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": container with ID starting with 757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264 not found: ID does not exist" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" Oct 31 00:10:01.797589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.797493 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264"} err="failed to get container status \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": rpc error: code = NotFound desc = could not find container \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": container with ID starting with 757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264 not found: ID does not exist" Oct 31 00:10:01.797589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.797517 5577 scope.go:122] "RemoveContainer" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" Oct 31 00:10:01.798182 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:01.798157 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": container with ID starting with fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf not found: ID does not exist" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" Oct 31 00:10:01.798304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.798211 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf"} err="failed to get container status \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": rpc error: code = NotFound desc = could not find container \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": container with ID starting with fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf not found: ID does not exist" Oct 31 00:10:01.798304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.798233 5577 scope.go:122] "RemoveContainer" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.799153 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b"} err="failed to get container status \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": rpc error: code = NotFound desc = could not find container \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": container with ID starting with c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b not found: ID does not exist" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.799177 5577 scope.go:122] "RemoveContainer" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.799435 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264"} err="failed to get container status \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": rpc error: code = NotFound desc = could not find container \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": container with ID starting with 757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264 not found: ID does not exist" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.799463 5577 scope.go:122] "RemoveContainer" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800300 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf"} err="failed to get container status \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": rpc error: code = NotFound desc = could not find container \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": container with ID starting with fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf not found: ID does not exist" Oct 31 00:10:01.800552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800326 5577 scope.go:122] "RemoveContainer" containerID="c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b" Oct 31 00:10:01.800965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800599 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b"} err="failed to get container status \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": rpc error: code = NotFound desc = could not find container \"c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b\": container with ID starting with c05b34f422f0503c344d7225cf45ecd0c7f62a46d0737b983241c8a915ca189b not found: ID does not exist" Oct 31 00:10:01.800965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800619 5577 scope.go:122] "RemoveContainer" containerID="757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264" Oct 31 00:10:01.800965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800873 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264"} err="failed to get container status \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": rpc error: code = NotFound desc = could not find container \"757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264\": container with ID starting with 757abfc079786591a27db99b9d77e09bfccbeeebebc5a168c0e8a2767b5f4264 not found: ID does not exist" Oct 31 00:10:01.800965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.800889 5577 scope.go:122] "RemoveContainer" containerID="fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf" Oct 31 00:10:01.801173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.801071 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf"} err="failed to get container status \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": rpc error: code = NotFound desc = could not find container \"fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf\": container with ID starting with fbc5e7992c73292066fd691b7e3a35446c9a1a884678ab8ae633b34f4e977bcf not found: ID does not exist" Oct 31 00:10:01.809538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.809514 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:01.809725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.809679 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" containerName="perses" containerID="cri-o://7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d" gracePeriod=30 Oct 31 00:10:01.827541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827517 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\") pod \"9561c21d-ebef-4f55-9309-01a542e3ad3b\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " Oct 31 00:10:01.827682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827647 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\" (UniqueName: \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\") pod \"9561c21d-ebef-4f55-9309-01a542e3ad3b\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827684 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\") pod \"9561c21d-ebef-4f55-9309-01a542e3ad3b\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827749 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\" (UniqueName: \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\") pod \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827820 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\") pod \"9561c21d-ebef-4f55-9309-01a542e3ad3b\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827856 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\") pod \"9561c21d-ebef-4f55-9309-01a542e3ad3b\" (UID: \"9561c21d-ebef-4f55-9309-01a542e3ad3b\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827901 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\" (UniqueName: \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\") pod \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " Oct 31 00:10:01.827976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.827959 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\" (UniqueName: \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\") pod \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\" (UID: \"5c07ac39-80a3-47b5-a338-0b8617fb4a39\") " Oct 31 00:10:01.828288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.828174 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca" pod "9561c21d-ebef-4f55-9309-01a542e3ad3b" (UID: "9561c21d-ebef-4f55-9309-01a542e3ad3b"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.828766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.828744 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config" pod "9561c21d-ebef-4f55-9309-01a542e3ad3b" (UID: "9561c21d-ebef-4f55-9309-01a542e3ad3b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.828919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.828841 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config" pod "5c07ac39-80a3-47b5-a338-0b8617fb4a39" (UID: "5c07ac39-80a3-47b5-a338-0b8617fb4a39"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.829459 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.829355 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca" pod "9561c21d-ebef-4f55-9309-01a542e3ad3b" (UID: "9561c21d-ebef-4f55-9309-01a542e3ad3b"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:01.832042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.832018 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft" pod "5c07ac39-80a3-47b5-a338-0b8617fb4a39" (UID: "5c07ac39-80a3-47b5-a338-0b8617fb4a39"). InnerVolumeSpecName "kube-api-access-l24ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:01.832042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.832033 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls" pod "9561c21d-ebef-4f55-9309-01a542e3ad3b" (UID: "9561c21d-ebef-4f55-9309-01a542e3ad3b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.832187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.832097 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr" pod "9561c21d-ebef-4f55-9309-01a542e3ad3b" (UID: "9561c21d-ebef-4f55-9309-01a542e3ad3b"). InnerVolumeSpecName "kube-api-access-7hmfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:01.832187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.832102 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls" pod "5c07ac39-80a3-47b5-a338-0b8617fb4a39" (UID: "5c07ac39-80a3-47b5-a338-0b8617fb4a39"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:01.845020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.844989 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:01.928990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.928945 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hmfr\" (UniqueName: \"kubernetes.io/projected/9561c21d-ebef-4f55-9309-01a542e3ad3b-kube-api-access-7hmfr\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.928990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.928981 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.928998 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l24ft\" (UniqueName: \"kubernetes.io/projected/5c07ac39-80a3-47b5-a338-0b8617fb4a39-kube-api-access-l24ft\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.929012 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/5c07ac39-80a3-47b5-a338-0b8617fb4a39-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.929025 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.929038 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9561c21d-ebef-4f55-9309-01a542e3ad3b-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.929051 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9561c21d-ebef-4f55-9309-01a542e3ad3b-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.929162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.929065 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/5c07ac39-80a3-47b5-a338-0b8617fb4a39-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:01.989257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:01.988876 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:02.035971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.035945 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:10:02.037888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.037864 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xvzmw"] Oct 31 00:10:02.050608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.050578 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:10:02.054345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.054320 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pjjp5"] Oct 31 00:10:02.117350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.117324 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:02.118744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.118699 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:02.130899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.130875 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:02.131034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.130902 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\" (UniqueName: \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\") pod \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " Oct 31 00:10:02.131034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.130983 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\" (UniqueName: \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\") pod \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " Oct 31 00:10:02.131151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.131111 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " Oct 31 00:10:02.131208 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.131153 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\" (UniqueName: \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\") pod \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\" (UID: \"c18e8ec3-6014-4f1d-b259-0b7737addcc6\") " Oct 31 00:10:02.131432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.131408 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config" pod "c18e8ec3-6014-4f1d-b259-0b7737addcc6" (UID: "c18e8ec3-6014-4f1d-b259-0b7737addcc6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.134991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.134608 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98" pod "c18e8ec3-6014-4f1d-b259-0b7737addcc6" (UID: "c18e8ec3-6014-4f1d-b259-0b7737addcc6"). InnerVolumeSpecName "kube-api-access-92n98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.140878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.140845 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "c18e8ec3-6014-4f1d-b259-0b7737addcc6" (UID: "c18e8ec3-6014-4f1d-b259-0b7737addcc6"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:10:02.232328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232241 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232306 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\" (UniqueName: \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232420 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232815 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232886 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232933 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\") pod \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\" (UID: \"393b10c8-c521-4b63-9e0d-a9e73853b7a9\") " Oct 31 00:10:02.232949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.232934 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:02.233272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.233152 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.233272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.233170 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c18e8ec3-6014-4f1d-b259-0b7737addcc6-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.233272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.233199 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:10:02.233272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.233217 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92n98\" (UniqueName: \"kubernetes.io/projected/c18e8ec3-6014-4f1d-b259-0b7737addcc6-kube-api-access-92n98\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.234974 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.234939 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.235301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.235273 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "kube-api-access-6sk24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:02.235470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.235447 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.235563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.235545 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.235732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.235686 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs" pod "393b10c8-c521-4b63-9e0d-a9e73853b7a9" (UID: "393b10c8-c521-4b63-9e0d-a9e73853b7a9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:02.247223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.247198 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:02.247789 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.247773 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:10:02.289995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.289961 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins" pod "c18e8ec3-6014-4f1d-b259-0b7737addcc6" (UID: "c18e8ec3-6014-4f1d-b259-0b7737addcc6"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:02.333920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333880 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.333920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333909 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.333920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333922 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.334153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333933 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:10:02.334153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333943 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/c18e8ec3-6014-4f1d-b259-0b7737addcc6-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.334153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333951 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/393b10c8-c521-4b63-9e0d-a9e73853b7a9-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.334153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.333958 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sk24\" (UniqueName: \"kubernetes.io/projected/393b10c8-c521-4b63-9e0d-a9e73853b7a9-kube-api-access-6sk24\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:02.720688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.720651 5577 generic.go:356] "Generic (PLEG): container finished" podID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" containerID="7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d" exitCode=0 Oct 31 00:10:02.721174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.720733 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:02.721174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.720751 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c18e8ec3-6014-4f1d-b259-0b7737addcc6","Type":"ContainerDied","Data":"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d"} Oct 31 00:10:02.721174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.720793 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"c18e8ec3-6014-4f1d-b259-0b7737addcc6","Type":"ContainerDied","Data":"52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034"} Oct 31 00:10:02.721174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.720808 5577 scope.go:122] "RemoveContainer" containerID="7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d" Oct 31 00:10:02.723424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.723396 5577 generic.go:356] "Generic (PLEG): container finished" podID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" containerID="9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a" exitCode=0 Oct 31 00:10:02.723536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.723446 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"393b10c8-c521-4b63-9e0d-a9e73853b7a9","Type":"ContainerDied","Data":"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a"} Oct 31 00:10:02.723536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.723467 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:02.723536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.723504 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"393b10c8-c521-4b63-9e0d-a9e73853b7a9","Type":"ContainerDied","Data":"884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4"} Oct 31 00:10:02.731294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.730839 5577 scope.go:122] "RemoveContainer" containerID="7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d" Oct 31 00:10:02.731294 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:02.731181 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d\": container with ID starting with 7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d not found: ID does not exist" containerID="7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d" Oct 31 00:10:02.731294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.731215 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d"} err="failed to get container status \"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d\": rpc error: code = NotFound desc = could not find container \"7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d\": container with ID starting with 7ccebd51e33956e0efa1328a80cc96966e9753918c8f1aea39d95c9d496e930d not found: ID does not exist" Oct 31 00:10:02.731294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.731236 5577 scope.go:122] "RemoveContainer" containerID="9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a" Oct 31 00:10:02.740095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.740074 5577 scope.go:122] "RemoveContainer" containerID="9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a" Oct 31 00:10:02.740428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:02.740401 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a\": container with ID starting with 9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a not found: ID does not exist" containerID="9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a" Oct 31 00:10:02.740505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.740439 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a"} err="failed to get container status \"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a\": rpc error: code = NotFound desc = could not find container \"9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a\": container with ID starting with 9e1c354ed5853113c6abeb0bb135195a8b5834709fdd8f562373b628110b714a not found: ID does not exist" Oct 31 00:10:02.748291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.748266 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:02.752729 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.752691 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:02.755303 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:02.755273 5577 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18e8ec3_6014_4f1d_b259_0b7737addcc6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b10c8_c521_4b63_9e0d_a9e73853b7a9.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc18e8ec3_6014_4f1d_b259_0b7737addcc6.slice/crio-52be7326146a814e38cce4126657b7e1b8f83615c7e32d95f3787bf38eb31034\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393b10c8_c521_4b63_9e0d_a9e73853b7a9.slice/crio-884462e49e088508d7f8e57d5b39d868c5c1111dab540e54a4e5263b5b6b88c4\": RecentStats: unable to find data in memory cache]" Oct 31 00:10:02.764988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.764964 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:02.769612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:02.769590 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:03.129076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:03.129044 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" path="/var/lib/kubelet/pods/393b10c8-c521-4b63-9e0d-a9e73853b7a9/volumes" Oct 31 00:10:03.129403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:03.129391 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" path="/var/lib/kubelet/pods/5c07ac39-80a3-47b5-a338-0b8617fb4a39/volumes" Oct 31 00:10:03.129813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:03.129802 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" path="/var/lib/kubelet/pods/9561c21d-ebef-4f55-9309-01a542e3ad3b/volumes" Oct 31 00:10:03.130529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:03.130515 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" path="/var/lib/kubelet/pods/c18e8ec3-6014-4f1d-b259-0b7737addcc6/volumes" Oct 31 00:10:06.531669 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:06.531630 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.38:9093/-/ready\": dial tcp 10.133.0.38:9093: connect: connection refused" Oct 31 00:10:09.460659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.460618 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:09.551293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551254 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:10:09.551673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551654 5577 cpu_manager.go:414] "containerMap: removing container" podUID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" containerName="otc-container" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551677 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" containerName="otc-container" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551698 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="method-gate" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551722 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="method-gate" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551739 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="kube-rbac-proxy" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551747 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="kube-rbac-proxy" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551756 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="prom-label-proxy" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551763 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="prom-label-proxy" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551773 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="method-gate" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551780 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="method-gate" Oct 31 00:10:09.551796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551795 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" containerName="perses" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551802 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" containerName="perses" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551833 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="kube-rbac-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551841 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="kube-rbac-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551851 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="prom-label-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551858 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="prom-label-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551936 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="393b10c8-c521-4b63-9e0d-a9e73853b7a9" containerName="otc-container" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551949 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="method-gate" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551958 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="kube-rbac-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551962 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07ac39-80a3-47b5-a338-0b8617fb4a39" containerName="prom-label-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551968 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="method-gate" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551974 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18e8ec3-6014-4f1d-b259-0b7737addcc6" containerName="perses" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551983 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="kube-rbac-proxy" Oct 31 00:10:09.552164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.551987 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9561c21d-ebef-4f55-9309-01a542e3ad3b" containerName="prom-label-proxy" Oct 31 00:10:09.555272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.555256 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.568943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.568916 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:10:09.574219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.574195 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:10:09.577614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.577597 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.589390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.589366 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:10:09.592289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592266 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.592396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592302 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m9p2\" (UniqueName: \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.592396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592358 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.592477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592415 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knb96\" (UniqueName: \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.592511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592489 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.592561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592544 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.592647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592635 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.592681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.592663 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.616098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.616078 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:09.649772 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649730 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:10:09.649906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649866 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:10:09.649906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649881 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:10:09.649906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649869 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:10:09.650032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649909 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:10:09.650032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649913 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:10:09.650032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649945 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-47xwz\"" Oct 31 00:10:09.650032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.649992 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-2nkl6\"" Oct 31 00:10:09.693337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693307 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.693460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693342 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.693460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693357 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m9p2\" (UniqueName: \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.693460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693400 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.693460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693427 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knb96\" (UniqueName: \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.693670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693458 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.693670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693500 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.693670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.693558 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.693670 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:09.693666 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:10:09.693882 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:09.693757 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls podName:76090f72-3199-4646-ab80-8809a4284b0f nodeName:}" failed. No retries permitted until 2026-10-31 00:10:10.193735678 +0000 UTC m=+649.185395254 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-z5htn" (UID: "76090f72-3199-4646-ab80-8809a4284b0f") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:10:09.694127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.694110 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.694165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.694130 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.694453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.694438 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.694576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.694554 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.696143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.696123 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.728885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.728818 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m9p2\" (UniqueName: \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-wbwrj\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.728991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.728944 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knb96\" (UniqueName: \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:09.748543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.748515 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:09.874733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.874685 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:10:09.894695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:09.894645 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.086146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.086118 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:10.090362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.090340 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.099265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.099238 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:10:10.099549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.099529 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:10:10.100399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.100374 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:10:10.100561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.100544 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:10:10.100628 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.100601 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:10:10.100737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.100685 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:10.122838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.122812 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:10.126252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.126231 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.136476 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.136458 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:10.156889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.156855 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:10:10.159961 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:10.159930 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda85f980_ec6f_4c5a_afc2_b84ed71ac9da.slice/crio-26aa62af96814baba023d6f1064caf9a44b1715af58db2d7b1652ac0b4c00273 WatchSource:0}: Error finding container 26aa62af96814baba023d6f1064caf9a44b1715af58db2d7b1652ac0b4c00273: Status 404 returned error can't find the container with id 26aa62af96814baba023d6f1064caf9a44b1715af58db2d7b1652ac0b4c00273 Oct 31 00:10:10.186316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.186292 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197315 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197347 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197367 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197400 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197419 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.197443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197435 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rpzs\" (UniqueName: \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197458 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197474 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m497w\" (UniqueName: \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.197784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197490 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.197784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.197513 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.199468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.199450 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:10:10.199556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.199482 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:10:10.200818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.200791 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-z5htn\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:10.298566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298543 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.298662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298576 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.298662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298611 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.298662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298632 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rpzs\" (UniqueName: \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.298867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298666 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.298867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298682 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m497w\" (UniqueName: \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.298867 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:10.298697 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:10:10.298867 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:10.298766 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs podName:01c25cef-0b59-4d77-a753-f92c0919ee9c nodeName:}" failed. No retries permitted until 2026-10-31 00:10:10.798750232 +0000 UTC m=+649.790409798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs") pod "data-science-collector-collector-0" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:10:10.299070 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298701 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.299070 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.298967 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.299070 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.299005 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.299343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.299319 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.299407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.299383 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.299407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.299389 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.301233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.301215 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.301347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.301326 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.301418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.301403 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.310962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.310938 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rpzs\" (UniqueName: \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.312429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.312406 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m497w\" (UniqueName: \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:10.316465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.316437 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.495265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.495230 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:10:10.558569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.558522 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:10.763390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.763357 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerStarted","Data":"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6"} Oct 31 00:10:10.763575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.763451 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerStarted","Data":"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4"} Oct 31 00:10:10.763575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.763480 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerStarted","Data":"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc"} Oct 31 00:10:10.763575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.763502 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerStarted","Data":"26aa62af96814baba023d6f1064caf9a44b1715af58db2d7b1652ac0b4c00273"} Oct 31 00:10:10.768325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.768298 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:10:10.773318 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:10.773290 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76090f72_3199_4646_ab80_8809a4284b0f.slice/crio-c228d1496a5db06b8ee0f776762605fdbaa02b2eb96b24ce8d246c4cd636b38e WatchSource:0}: Error finding container c228d1496a5db06b8ee0f776762605fdbaa02b2eb96b24ce8d246c4cd636b38e: Status 404 returned error can't find the container with id c228d1496a5db06b8ee0f776762605fdbaa02b2eb96b24ce8d246c4cd636b38e Oct 31 00:10:10.785205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.785165 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" podStartSLOduration=1.785150741 podStartE2EDuration="1.785150741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:10.783994566 +0000 UTC m=+649.775654159" watchObservedRunningTime="2026-10-31 00:10:10.785150741 +0000 UTC m=+649.776810333" Oct 31 00:10:10.804653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.803936 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.806773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.806753 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:10.952412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:10.952383 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:11.009088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.009066 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:11.171185 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.171153 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:11.285506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.285306 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:11.288282 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:11.288251 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c25cef_0b59_4d77_a753_f92c0919ee9c.slice/crio-8309ad4332ed775f3d35af7237b6c8d767cd50a8f51e5e2e4178994e0cf082a7 WatchSource:0}: Error finding container 8309ad4332ed775f3d35af7237b6c8d767cd50a8f51e5e2e4178994e0cf082a7: Status 404 returned error can't find the container with id 8309ad4332ed775f3d35af7237b6c8d767cd50a8f51e5e2e4178994e0cf082a7 Oct 31 00:10:11.291213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.291175 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:11.291296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.291255 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:11.530987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.530950 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.38:9093/-/ready\": dial tcp 10.133.0.38:9093: connect: connection refused" Oct 31 00:10:11.772523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.772486 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"01c25cef-0b59-4d77-a753-f92c0919ee9c","Type":"ContainerStarted","Data":"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0"} Oct 31 00:10:11.772698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.772619 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"01c25cef-0b59-4d77-a753-f92c0919ee9c","Type":"ContainerStarted","Data":"8309ad4332ed775f3d35af7237b6c8d767cd50a8f51e5e2e4178994e0cf082a7"} Oct 31 00:10:11.775568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.775538 5577 generic.go:356] "Generic (PLEG): container finished" podID="28a27225-917d-456f-8c9b-92e0d6264853" containerID="b45f650259e79e0e7b430bd610c72645d28daaddfa70a65ccc81a48973d39955" exitCode=0 Oct 31 00:10:11.775748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.775700 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerDied","Data":"b45f650259e79e0e7b430bd610c72645d28daaddfa70a65ccc81a48973d39955"} Oct 31 00:10:11.778126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.778100 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerStarted","Data":"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6"} Oct 31 00:10:11.778232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.778172 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerStarted","Data":"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4"} Oct 31 00:10:11.778232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.778213 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerStarted","Data":"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4"} Oct 31 00:10:11.778324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.778254 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerStarted","Data":"c228d1496a5db06b8ee0f776762605fdbaa02b2eb96b24ce8d246c4cd636b38e"} Oct 31 00:10:11.793379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.793331 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.793316145 podStartE2EDuration="1.793316145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.793199255 +0000 UTC m=+650.784858852" watchObservedRunningTime="2026-10-31 00:10:11.793316145 +0000 UTC m=+650.784975731" Oct 31 00:10:11.813599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.813546 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" podStartSLOduration=2.813528726 podStartE2EDuration="2.813528726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.812752574 +0000 UTC m=+650.804412187" watchObservedRunningTime="2026-10-31 00:10:11.813528726 +0000 UTC m=+650.805188311" Oct 31 00:10:11.828629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.828609 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:11.913649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913577 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913618 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913646 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913674 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913701 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913785 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.913950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.913828 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\") pod \"28a27225-917d-456f-8c9b-92e0d6264853\" (UID: \"28a27225-917d-456f-8c9b-92e0d6264853\") " Oct 31 00:10:11.914225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.914176 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:11.916013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.915985 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:11.916447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.916425 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "kube-api-access-5w7ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:11.916704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.916679 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:10:11.916796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.916689 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:11.920131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.920108 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:11.926802 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:11.926780 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config" pod "28a27225-917d-456f-8c9b-92e0d6264853" (UID: "28a27225-917d-456f-8c9b-92e0d6264853"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:12.014537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014510 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014534 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014544 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/28a27225-917d-456f-8c9b-92e0d6264853-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014552 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014560 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w7ft\" (UniqueName: \"kubernetes.io/projected/28a27225-917d-456f-8c9b-92e0d6264853-kube-api-access-5w7ft\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014569 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.014694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.014579 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/28a27225-917d-456f-8c9b-92e0d6264853-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:12.785034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.785001 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:12.785034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.785016 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"28a27225-917d-456f-8c9b-92e0d6264853","Type":"ContainerDied","Data":"7ffd6b5886840b43a5015767170e469f3bc0eb49171c8a8e183caa893dc3d9b2"} Oct 31 00:10:12.785478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.785081 5577 scope.go:122] "RemoveContainer" containerID="1678b244b09ded185925d459a181160ee24991d16e7c7708a47255adc648b0ec" Oct 31 00:10:12.794019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.794002 5577 scope.go:122] "RemoveContainer" containerID="b45f650259e79e0e7b430bd610c72645d28daaddfa70a65ccc81a48973d39955" Oct 31 00:10:12.801643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.801625 5577 scope.go:122] "RemoveContainer" containerID="5de30251cf85f50973a1a9c4498f30d121bd9781904faa803cfc0a0fab5df2a3" Oct 31 00:10:12.807780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.807758 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:12.813702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:12.813681 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:13.129448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:13.129416 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a27225-917d-456f-8c9b-92e0d6264853" path="/var/lib/kubelet/pods/28a27225-917d-456f-8c9b-92e0d6264853/volumes" Oct 31 00:10:15.438649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.438615 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.447425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.447397 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.539662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.539627 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.539826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.539780 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.542731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.542699 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.632817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.632784 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.642734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.642675 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:15.651738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.651694 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.841919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.841891 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:15.865232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.865179 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:15.997870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:15.997839 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:16.118284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.118198 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:10:16.122171 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:16.122142 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddfadc60c_8b53_4331_91d7_7b131492017f.slice/crio-91e24faf4d3ce595199e78f6304d0685493635061d32b77799b3b259a4a05d71 WatchSource:0}: Error finding container 91e24faf4d3ce595199e78f6304d0685493635061d32b77799b3b259a4a05d71: Status 404 returned error can't find the container with id 91e24faf4d3ce595199e78f6304d0685493635061d32b77799b3b259a4a05d71 Oct 31 00:10:16.147078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.147042 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:16.148427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148398 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:16.148841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148819 5577 cpu_manager.go:414] "containerMap: removing container" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="config-reloader" Oct 31 00:10:16.148841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148841 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="config-reloader" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148863 5577 cpu_manager.go:414] "containerMap: removing container" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148874 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148903 5577 cpu_manager.go:414] "containerMap: removing container" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="init-config-reloader" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148911 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="init-config-reloader" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148988 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="config-reloader" Oct 31 00:10:16.148996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.148997 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a27225-917d-456f-8c9b-92e0d6264853" containerName="alertmanager" Oct 31 00:10:16.153884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.153861 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.165744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.165676 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:16.234811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.234775 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:10:16.234811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.234799 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:10:16.234811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.234775 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:10:16.235019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.234843 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-qnf9d\"" Oct 31 00:10:16.235019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.234844 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:10:16.245377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245347 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245380 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245401 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245423 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245525 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245550 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdnc6\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.245823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.245625 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346699 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346773 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346815 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346832 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdnc6\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346871 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346917 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.346940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.346932 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.347258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.347232 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.349572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.349540 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.349864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.349841 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.350071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.350055 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.350814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.350791 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.351310 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.351294 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.360129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.360097 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdnc6\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.489679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.489593 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:16.782260 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.782218 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:10:16.787835 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:16.787806 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b9d0e9_34b0_45d6_a8ba_8f6da9f92947.slice/crio-83b21b9975443902f221d44cb8e09b76e1fe044f3edff619f8f7a378fd7a00f7 WatchSource:0}: Error finding container 83b21b9975443902f221d44cb8e09b76e1fe044f3edff619f8f7a378fd7a00f7: Status 404 returned error can't find the container with id 83b21b9975443902f221d44cb8e09b76e1fe044f3edff619f8f7a378fd7a00f7 Oct 31 00:10:16.812086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.811996 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"dfadc60c-8b53-4331-91d7-7b131492017f","Type":"ContainerStarted","Data":"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d"} Oct 31 00:10:16.812086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.812050 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:16.812086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.812067 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"dfadc60c-8b53-4331-91d7-7b131492017f","Type":"ContainerStarted","Data":"91e24faf4d3ce595199e78f6304d0685493635061d32b77799b3b259a4a05d71"} Oct 31 00:10:16.813302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.813274 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerStarted","Data":"83b21b9975443902f221d44cb8e09b76e1fe044f3edff619f8f7a378fd7a00f7"} Oct 31 00:10:16.830760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:16.830676 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.830627154 podStartE2EDuration="6.830627154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:16.830030684 +0000 UTC m=+655.821690268" watchObservedRunningTime="2026-10-31 00:10:16.830627154 +0000 UTC m=+655.822286738" Oct 31 00:10:17.819775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:17.819735 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerStarted","Data":"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f"} Oct 31 00:10:20.833722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:20.833672 5577 generic.go:356] "Generic (PLEG): container finished" podID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerID="34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f" exitCode=0 Oct 31 00:10:20.834083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:20.833750 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerDied","Data":"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f"} Oct 31 00:10:21.840657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:21.840626 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerStarted","Data":"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8"} Oct 31 00:10:21.841091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:21.840688 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerStarted","Data":"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2"} Oct 31 00:10:21.841091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:21.840893 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:21.867900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:21.867842 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.277159427 podStartE2EDuration="5.867829316s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.590669889s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:21.866396185 +0000 UTC m=+660.858055796" watchObservedRunningTime="2026-10-31 00:10:21.867829316 +0000 UTC m=+660.859488901" Oct 31 00:10:25.089159 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:25.089124 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:25.220334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:25.220306 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:27.822917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:27.822887 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:10:27.845835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:27.845809 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:10:29.292357 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:29.292313 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:10:47.759287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:47.759253 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:47.958073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:47.958040 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:48.402390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:48.402355 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:48.538861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:48.538826 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:10:51.571580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:51.571544 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:51.572369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:51.572334 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="01c25cef-0b59-4d77-a753-f92c0919ee9c" containerName="otc-container" containerID="cri-o://2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0" gracePeriod=30 Oct 31 00:10:52.809077 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.809048 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:52.964182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964093 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964129 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964167 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964202 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\" (UniqueName: \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964242 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964287 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\") pod \"01c25cef-0b59-4d77-a753-f92c0919ee9c\" (UID: \"01c25cef-0b59-4d77-a753-f92c0919ee9c\") " Oct 31 00:10:52.964780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.964639 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:52.966563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.966528 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:52.966682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.966560 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:52.966682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.966654 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "kube-api-access-4rpzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:52.966964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.966935 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:52.966964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.966946 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs" pod "01c25cef-0b59-4d77-a753-f92c0919ee9c" (UID: "01c25cef-0b59-4d77-a753-f92c0919ee9c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:52.974609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.974582 5577 generic.go:356] "Generic (PLEG): container finished" podID="01c25cef-0b59-4d77-a753-f92c0919ee9c" containerID="2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0" exitCode=0 Oct 31 00:10:52.974736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.974720 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:52.974811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.974700 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"01c25cef-0b59-4d77-a753-f92c0919ee9c","Type":"ContainerDied","Data":"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0"} Oct 31 00:10:52.974891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.974876 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"01c25cef-0b59-4d77-a753-f92c0919ee9c","Type":"ContainerDied","Data":"8309ad4332ed775f3d35af7237b6c8d767cd50a8f51e5e2e4178994e0cf082a7"} Oct 31 00:10:52.974943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.974901 5577 scope.go:122] "RemoveContainer" containerID="2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0" Oct 31 00:10:52.984233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.984218 5577 scope.go:122] "RemoveContainer" containerID="2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0" Oct 31 00:10:52.984505 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:52.984487 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0\": container with ID starting with 2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0 not found: ID does not exist" containerID="2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0" Oct 31 00:10:52.984568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.984512 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0"} err="failed to get container status \"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0\": rpc error: code = NotFound desc = could not find container \"2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0\": container with ID starting with 2157373b1dc4094cf6aafc5efff189f3bbfdd199bdf4f7063773adca7d0580e0 not found: ID does not exist" Oct 31 00:10:52.995040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.995013 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:52.997227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:52.997205 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:53.023038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.023015 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:53.023337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.023325 5577 cpu_manager.go:414] "containerMap: removing container" podUID="01c25cef-0b59-4d77-a753-f92c0919ee9c" containerName="otc-container" Oct 31 00:10:53.023377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.023339 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c25cef-0b59-4d77-a753-f92c0919ee9c" containerName="otc-container" Oct 31 00:10:53.023421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.023407 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c25cef-0b59-4d77-a753-f92c0919ee9c" containerName="otc-container" Oct 31 00:10:53.027581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.027567 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.031559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.031537 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:10:53.031660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.031645 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:10:53.031756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.031679 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:10:53.031812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.031755 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:10:53.031812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.031755 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:10:53.032103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.032090 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:10:53.038952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.038927 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:53.065585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065563 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.065585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065581 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.065728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065592 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.065728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065601 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/01c25cef-0b59-4d77-a753-f92c0919ee9c-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.065728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065610 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/01c25cef-0b59-4d77-a753-f92c0919ee9c-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.065728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.065618 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rpzs\" (UniqueName: \"kubernetes.io/projected/01c25cef-0b59-4d77-a753-f92c0919ee9c-kube-api-access-4rpzs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:53.129697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.129657 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c25cef-0b59-4d77-a753-f92c0919ee9c" path="/var/lib/kubelet/pods/01c25cef-0b59-4d77-a753-f92c0919ee9c/volumes" Oct 31 00:10:53.166903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.166872 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.167045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.166910 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjml\" (UniqueName: \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.167045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.166932 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.167045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.167021 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.167172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.167095 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.167172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.167127 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.267915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.267876 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.267915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.267917 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.268113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.267967 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.268113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.267994 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjml\" (UniqueName: \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.268113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.268016 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.268113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.268078 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.268657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.268632 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.270869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.270847 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.270977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.270847 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.271147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.271128 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.271187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.271175 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.280192 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.280173 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjml\" (UniqueName: \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\") pod \"data-science-collector-collector-0\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.340300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.340277 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:10:53.615233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.615153 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:10:53.619376 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:10:53.619350 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52798cce_fca2_4274_9c37_308fc913a74b.slice/crio-4bcddfa666b4e07facc075ba3370aba675c55c9350eda75333921a425e4167fa WatchSource:0}: Error finding container 4bcddfa666b4e07facc075ba3370aba675c55c9350eda75333921a425e4167fa: Status 404 returned error can't find the container with id 4bcddfa666b4e07facc075ba3370aba675c55c9350eda75333921a425e4167fa Oct 31 00:10:53.621581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.621546 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:53.621668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.621604 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:10:53.980951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.980838 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52798cce-fca2-4274-9c37-308fc913a74b","Type":"ContainerStarted","Data":"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91"} Oct 31 00:10:53.980951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:53.980904 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52798cce-fca2-4274-9c37-308fc913a74b","Type":"ContainerStarted","Data":"4bcddfa666b4e07facc075ba3370aba675c55c9350eda75333921a425e4167fa"} Oct 31 00:10:54.001536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:10:54.001480 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.001464826 podStartE2EDuration="1.001464826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:54.001361338 +0000 UTC m=+692.993020953" watchObservedRunningTime="2026-10-31 00:10:54.001464826 +0000 UTC m=+692.993124407" Oct 31 00:10:59.291530 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:10:59.291495 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:11:04.244165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:11:04.244118 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:04.366413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:11:04.366377 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:25.170646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:11:25.170609 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:11:29.291697 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:11:29.291625 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:11:59.291948 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:11:59.291911 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:12:16.148737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:16.148677 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:16.287160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:16.287123 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:16.847220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:16.847164 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:16.973568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:16.973532 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:18.486935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:18.486895 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:18.615123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:18.615087 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:21.013406 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:21.013373 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:21.013911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:21.013687 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="52798cce-fca2-4274-9c37-308fc913a74b" containerName="otc-container" containerID="cri-o://1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91" gracePeriod=30 Oct 31 00:12:23.173627 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:23.173591 5577 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52798cce_fca2_4274_9c37_308fc913a74b.slice/crio-conmon-1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52798cce_fca2_4274_9c37_308fc913a74b.slice/crio-1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:12:23.364095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.364067 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:23.365193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.365167 5577 generic.go:356] "Generic (PLEG): container finished" podID="52798cce-fca2-4274-9c37-308fc913a74b" containerID="1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91" exitCode=0 Oct 31 00:12:23.365344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.365233 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52798cce-fca2-4274-9c37-308fc913a74b","Type":"ContainerDied","Data":"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91"} Oct 31 00:12:23.365344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.365278 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"52798cce-fca2-4274-9c37-308fc913a74b","Type":"ContainerDied","Data":"4bcddfa666b4e07facc075ba3370aba675c55c9350eda75333921a425e4167fa"} Oct 31 00:12:23.365344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.365299 5577 scope.go:122] "RemoveContainer" containerID="1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91" Oct 31 00:12:23.376280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.376246 5577 scope.go:122] "RemoveContainer" containerID="1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91" Oct 31 00:12:23.376936 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:23.376913 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91\": container with ID starting with 1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91 not found: ID does not exist" containerID="1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91" Oct 31 00:12:23.377047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.376944 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91"} err="failed to get container status \"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91\": rpc error: code = NotFound desc = could not find container \"1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91\": container with ID starting with 1553fa2319be2bd59b69aa6a0c6ec9a5514d712515714f9e130a5ba4f80fcd91 not found: ID does not exist" Oct 31 00:12:23.414055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.413992 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414044 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414084 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414109 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414132 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414182 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\" (UniqueName: \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\") pod \"52798cce-fca2-4274-9c37-308fc913a74b\" (UID: \"52798cce-fca2-4274-9c37-308fc913a74b\") " Oct 31 00:12:23.414515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.414472 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:23.416312 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.416284 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:23.416921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.416553 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:23.417024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.416939 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:23.417024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.416960 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:23.423320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.423265 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml" pod "52798cce-fca2-4274-9c37-308fc913a74b" (UID: "52798cce-fca2-4274-9c37-308fc913a74b"). InnerVolumeSpecName "kube-api-access-bgjml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:23.515531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515497 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:23.515531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515528 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:23.515702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515542 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/52798cce-fca2-4274-9c37-308fc913a74b-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:23.515702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515556 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:23.515702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515568 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/52798cce-fca2-4274-9c37-308fc913a74b-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:23.515702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:23.515580 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjml\" (UniqueName: \"kubernetes.io/projected/52798cce-fca2-4274-9c37-308fc913a74b-kube-api-access-bgjml\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:24.371658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.371629 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.395523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.395499 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:24.400161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.400139 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:24.426692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.426673 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:24.427008 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.426996 5577 cpu_manager.go:414] "containerMap: removing container" podUID="52798cce-fca2-4274-9c37-308fc913a74b" containerName="otc-container" Oct 31 00:12:24.427050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.427010 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52798cce-fca2-4274-9c37-308fc913a74b" containerName="otc-container" Oct 31 00:12:24.427085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.427076 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="52798cce-fca2-4274-9c37-308fc913a74b" containerName="otc-container" Oct 31 00:12:24.431261 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.431247 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.434987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.434966 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:12:24.435594 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.435578 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:24.435785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.435767 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:24.435785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.435780 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:24.435892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.435776 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:24.435892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.435772 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:24.439990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.439969 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:24.524389 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524360 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.524518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524432 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.524518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524498 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p8zg\" (UniqueName: \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.524615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524521 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.524615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524571 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.524695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.524630 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625016 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625052 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625071 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625130 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p8zg\" (UniqueName: \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625161 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625197 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.625897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.625872 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.627692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.627673 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.627816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.627763 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.628173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.628156 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.628235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.628183 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.637943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.637916 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p8zg\" (UniqueName: \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\") pod \"data-science-collector-collector-0\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:24.744257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:24.744226 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:25.029822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.029801 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:25.032328 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:12:25.032294 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43c2f3d2_0b04_497a_9383_55e583b48eb8.slice/crio-ca71c25abadced811e65761450374d1f14b74740691632f8f15e3f9e2ca88207 WatchSource:0}: Error finding container ca71c25abadced811e65761450374d1f14b74740691632f8f15e3f9e2ca88207: Status 404 returned error can't find the container with id ca71c25abadced811e65761450374d1f14b74740691632f8f15e3f9e2ca88207 Oct 31 00:12:25.034443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.034412 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:25.034509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.034471 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:25.129914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.129872 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52798cce-fca2-4274-9c37-308fc913a74b" path="/var/lib/kubelet/pods/52798cce-fca2-4274-9c37-308fc913a74b/volumes" Oct 31 00:12:25.207464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.207411 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:25.378056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.377967 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43c2f3d2-0b04-497a-9383-55e583b48eb8","Type":"ContainerStarted","Data":"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba"} Oct 31 00:12:25.378056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.378027 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43c2f3d2-0b04-497a-9383-55e583b48eb8","Type":"ContainerStarted","Data":"ca71c25abadced811e65761450374d1f14b74740691632f8f15e3f9e2ca88207"} Oct 31 00:12:25.398927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:25.398875 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.398861113 podStartE2EDuration="1.398861113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:25.398030309 +0000 UTC m=+784.389689894" watchObservedRunningTime="2026-10-31 00:12:25.398861113 +0000 UTC m=+784.390520696" Oct 31 00:12:28.764663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:28.764626 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:28.899614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:28.899579 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:29.293055 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:29.293019 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:12:29.372546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:29.372514 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:29.509810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:29.509770 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:29.645372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:29.645292 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:31.064119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.064091 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:31.064699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.064263 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="43c2f3d2-0b04-497a-9383-55e583b48eb8" containerName="otc-container" containerID="cri-o://4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba" gracePeriod=30 Oct 31 00:12:31.299816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.299786 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.388182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388109 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.388182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388147 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\" (UniqueName: \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.388182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388171 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.388412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388229 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.388412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388283 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.388412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.388336 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\") pod \"43c2f3d2-0b04-497a-9383-55e583b48eb8\" (UID: \"43c2f3d2-0b04-497a-9383-55e583b48eb8\") " Oct 31 00:12:31.391217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.391186 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:31.393017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.392983 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:31.393145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.393099 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:31.393418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.393391 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:31.393530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.393461 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "kube-api-access-7p8zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:31.393530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.393464 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs" pod "43c2f3d2-0b04-497a-9383-55e583b48eb8" (UID: "43c2f3d2-0b04-497a-9383-55e583b48eb8"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:31.405818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.405789 5577 generic.go:356] "Generic (PLEG): container finished" podID="43c2f3d2-0b04-497a-9383-55e583b48eb8" containerID="4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba" exitCode=0 Oct 31 00:12:31.405914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.405831 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43c2f3d2-0b04-497a-9383-55e583b48eb8","Type":"ContainerDied","Data":"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba"} Oct 31 00:12:31.405914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.405844 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.405914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.405891 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"43c2f3d2-0b04-497a-9383-55e583b48eb8","Type":"ContainerDied","Data":"ca71c25abadced811e65761450374d1f14b74740691632f8f15e3f9e2ca88207"} Oct 31 00:12:31.406038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.405920 5577 scope.go:122] "RemoveContainer" containerID="4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba" Oct 31 00:12:31.414748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.414734 5577 scope.go:122] "RemoveContainer" containerID="4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba" Oct 31 00:12:31.415001 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:31.414979 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba\": container with ID starting with 4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba not found: ID does not exist" containerID="4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba" Oct 31 00:12:31.415065 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.415013 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba"} err="failed to get container status \"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba\": rpc error: code = NotFound desc = could not find container \"4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba\": container with ID starting with 4ec57d8c3a7cfaa63d9bd164f6056a36526eb154fe6c92ce423150f0e530d2ba not found: ID does not exist" Oct 31 00:12:31.426410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.426388 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:31.430182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.430165 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:31.457902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.457872 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:31.458230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.458217 5577 cpu_manager.go:414] "containerMap: removing container" podUID="43c2f3d2-0b04-497a-9383-55e583b48eb8" containerName="otc-container" Oct 31 00:12:31.458271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.458232 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43c2f3d2-0b04-497a-9383-55e583b48eb8" containerName="otc-container" Oct 31 00:12:31.458323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.458314 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="43c2f3d2-0b04-497a-9383-55e583b48eb8" containerName="otc-container" Oct 31 00:12:31.462852 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.462837 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.467275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467254 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:31.467445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467280 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:31.467445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467297 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:12:31.467445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467327 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:31.467445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467365 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:31.467445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.467365 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:31.471381 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.471360 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:31.489015 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.488995 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/43c2f3d2-0b04-497a-9383-55e583b48eb8-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.489015 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.489015 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.489202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.489024 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p8zg\" (UniqueName: \"kubernetes.io/projected/43c2f3d2-0b04-497a-9383-55e583b48eb8-kube-api-access-7p8zg\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.489202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.489034 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.489202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.489047 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.489202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.489058 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/43c2f3d2-0b04-497a-9383-55e583b48eb8-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:31.589781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.589740 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.589899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.589796 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.589949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.589898 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.589949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.589933 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.590021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.589956 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z76jx\" (UniqueName: \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.590067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.590053 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690678 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690593 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690678 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690657 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690696 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690755 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690794 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.690915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.690814 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z76jx\" (UniqueName: \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.691511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.691482 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.693502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.693466 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.693598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.693499 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.693883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.693865 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.693951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.693934 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.702956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.702936 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z76jx\" (UniqueName: \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\") pod \"data-science-collector-collector-0\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:31.776786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:31.776765 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:32.048001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.047815 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:32.050557 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:12:32.050528 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e206252_ea05_48c6_91b3_2aaf0c4d46da.slice/crio-d93b908c2b7d17bf05c634e7347cb7e331e945e1d383bd43b8c0b8d5e769b1d1 WatchSource:0}: Error finding container d93b908c2b7d17bf05c634e7347cb7e331e945e1d383bd43b8c0b8d5e769b1d1: Status 404 returned error can't find the container with id d93b908c2b7d17bf05c634e7347cb7e331e945e1d383bd43b8c0b8d5e769b1d1 Oct 31 00:12:32.052659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.052628 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:32.052773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.052686 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:32.412991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.412906 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e206252-ea05-48c6-91b3-2aaf0c4d46da","Type":"ContainerStarted","Data":"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c"} Oct 31 00:12:32.412991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.412954 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e206252-ea05-48c6-91b3-2aaf0c4d46da","Type":"ContainerStarted","Data":"d93b908c2b7d17bf05c634e7347cb7e331e945e1d383bd43b8c0b8d5e769b1d1"} Oct 31 00:12:32.433420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:32.433361 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.433342752 podStartE2EDuration="1.433342752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:32.432398346 +0000 UTC m=+791.424057942" watchObservedRunningTime="2026-10-31 00:12:32.433342752 +0000 UTC m=+791.425002337" Oct 31 00:12:33.129427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:33.129393 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43c2f3d2-0b04-497a-9383-55e583b48eb8" path="/var/lib/kubelet/pods/43c2f3d2-0b04-497a-9383-55e583b48eb8/volumes" Oct 31 00:12:41.285982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:41.285939 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:41.435480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:41.435443 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:41.938532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:41.938499 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:42.076105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:42.076044 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:43.191759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:43.191672 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:43.444024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:43.443937 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:46.146859 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.146829 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:46.147255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.147018 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" containerName="otc-container" containerID="cri-o://afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c" gracePeriod=30 Oct 31 00:12:46.391068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.391045 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.470897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.470815 5577 generic.go:356] "Generic (PLEG): container finished" podID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" containerID="afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c" exitCode=0 Oct 31 00:12:46.470897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.470867 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e206252-ea05-48c6-91b3-2aaf0c4d46da","Type":"ContainerDied","Data":"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c"} Oct 31 00:12:46.470897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.470882 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.471119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.470921 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"2e206252-ea05-48c6-91b3-2aaf0c4d46da","Type":"ContainerDied","Data":"d93b908c2b7d17bf05c634e7347cb7e331e945e1d383bd43b8c0b8d5e769b1d1"} Oct 31 00:12:46.471119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.470945 5577 scope.go:122] "RemoveContainer" containerID="afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c" Oct 31 00:12:46.487951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.487927 5577 scope.go:122] "RemoveContainer" containerID="afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c" Oct 31 00:12:46.488278 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:46.488256 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c\": container with ID starting with afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c not found: ID does not exist" containerID="afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c" Oct 31 00:12:46.488329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.488286 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c"} err="failed to get container status \"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c\": rpc error: code = NotFound desc = could not find container \"afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c\": container with ID starting with afebb9d76fed0b7ec97d3dacf7723faecac0af7ebbeab20b534c26af57fc112c not found: ID does not exist" Oct 31 00:12:46.521407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521379 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.521586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521421 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.521586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521474 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.521586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521499 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.521781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521593 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\" (UniqueName: \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.521781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.521611 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\") pod \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\" (UID: \"2e206252-ea05-48c6-91b3-2aaf0c4d46da\") " Oct 31 00:12:46.522064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.522035 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:46.523917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.523877 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:46.524372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.524344 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:46.524472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.524392 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "kube-api-access-z76jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:46.524472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.524401 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:46.524472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.524417 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs" pod "2e206252-ea05-48c6-91b3-2aaf0c4d46da" (UID: "2e206252-ea05-48c6-91b3-2aaf0c4d46da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:46.622909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622880 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z76jx\" (UniqueName: \"kubernetes.io/projected/2e206252-ea05-48c6-91b3-2aaf0c4d46da-kube-api-access-z76jx\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.622909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622905 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.622909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622917 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.623125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622927 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.623125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622936 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e206252-ea05-48c6-91b3-2aaf0c4d46da-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.623125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.622945 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e206252-ea05-48c6-91b3-2aaf0c4d46da-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:46.793608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.793580 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:46.797686 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.797658 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:46.824315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.824287 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:46.824689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.824676 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" containerName="otc-container" Oct 31 00:12:46.824766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.824690 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" containerName="otc-container" Oct 31 00:12:46.824809 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.824789 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" containerName="otc-container" Oct 31 00:12:46.829288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.829273 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.832919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.832899 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:46.835351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.835329 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:46.835987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.835876 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 31 00:12:46.838046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.838023 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:46.925365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925333 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.925533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925374 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.925533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925398 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.925627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925534 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.925627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925572 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.925693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.925637 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mdtx\" (UniqueName: \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:46.933864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.933830 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:46.933864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.933833 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:46.933864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:46.933857 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:47.026043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026014 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.026043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026050 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.026298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026082 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.026298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026108 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.026298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026140 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mdtx\" (UniqueName: \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.026298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.026290 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.027165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.027134 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.028927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.028871 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.029038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.028973 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.029038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.028986 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.029038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.028987 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.041494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.041470 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mdtx\" (UniqueName: \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\") pod \"data-science-collector-collector-0\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.129483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.129447 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e206252-ea05-48c6-91b3-2aaf0c4d46da" path="/var/lib/kubelet/pods/2e206252-ea05-48c6-91b3-2aaf0c4d46da/volumes" Oct 31 00:12:47.143059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.143035 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:47.421952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.421924 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:47.424440 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:12:47.424411 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53232afd_8742_4add_b568_05b9d17b4a4b.slice/crio-35b14ddac5a16edd55b01ca998a2bf081de17c3482bc96eda68350278a4df174 WatchSource:0}: Error finding container 35b14ddac5a16edd55b01ca998a2bf081de17c3482bc96eda68350278a4df174: Status 404 returned error can't find the container with id 35b14ddac5a16edd55b01ca998a2bf081de17c3482bc96eda68350278a4df174 Oct 31 00:12:47.426838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.426803 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:47.426933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.426880 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:47.475787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:47.475754 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53232afd-8742-4add-b568-05b9d17b4a4b","Type":"ContainerStarted","Data":"35b14ddac5a16edd55b01ca998a2bf081de17c3482bc96eda68350278a4df174"} Oct 31 00:12:48.483092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:48.483056 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53232afd-8742-4add-b568-05b9d17b4a4b","Type":"ContainerStarted","Data":"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea"} Oct 31 00:12:48.504339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:48.504293 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.504278987 podStartE2EDuration="2.504278987s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:48.50365186 +0000 UTC m=+807.495311458" watchObservedRunningTime="2026-10-31 00:12:48.504278987 +0000 UTC m=+807.495938565" Oct 31 00:12:53.507488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:53.507451 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:53.642703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:53.642667 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:54.252887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:54.252854 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:54.447632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:54.447600 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:54.613192 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:54.613102 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:12:56.237653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.237623 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:56.238089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.237836 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="53232afd-8742-4add-b568-05b9d17b4a4b" containerName="otc-container" containerID="cri-o://169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea" gracePeriod=30 Oct 31 00:12:56.488543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.488481 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:56.523263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.523232 5577 generic.go:356] "Generic (PLEG): container finished" podID="53232afd-8742-4add-b568-05b9d17b4a4b" containerID="169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea" exitCode=0 Oct 31 00:12:56.523263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.523264 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:56.523542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.523291 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53232afd-8742-4add-b568-05b9d17b4a4b","Type":"ContainerDied","Data":"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea"} Oct 31 00:12:56.523542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.523336 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"53232afd-8742-4add-b568-05b9d17b4a4b","Type":"ContainerDied","Data":"35b14ddac5a16edd55b01ca998a2bf081de17c3482bc96eda68350278a4df174"} Oct 31 00:12:56.523542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.523359 5577 scope.go:122] "RemoveContainer" containerID="169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea" Oct 31 00:12:56.532080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.532060 5577 scope.go:122] "RemoveContainer" containerID="169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea" Oct 31 00:12:56.532385 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:56.532366 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea\": container with ID starting with 169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea not found: ID does not exist" containerID="169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea" Oct 31 00:12:56.532456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.532400 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea"} err="failed to get container status \"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea\": rpc error: code = NotFound desc = could not find container \"169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea\": container with ID starting with 169c54aa89d2a33d0fb1ca028e6bac812038c2c44f7ec11998f478235a3215ea not found: ID does not exist" Oct 31 00:12:56.614225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614184 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\" (UniqueName: \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614284 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614320 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614352 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614420 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614475 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\") pod \"53232afd-8742-4add-b568-05b9d17b4a4b\" (UID: \"53232afd-8742-4add-b568-05b9d17b4a4b\") " Oct 31 00:12:56.614802 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.614773 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:12:56.616666 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.616633 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:56.616825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.616740 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:56.616985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.616961 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:56.617050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.617033 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "kube-api-access-7mdtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:12:56.617165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.617149 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert" pod "53232afd-8742-4add-b568-05b9d17b4a4b" (UID: "53232afd-8742-4add-b568-05b9d17b4a4b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:12:56.715663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715630 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.715663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715661 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.715663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715672 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mdtx\" (UniqueName: \"kubernetes.io/projected/53232afd-8742-4add-b568-05b9d17b4a4b-kube-api-access-7mdtx\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.715922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715682 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/53232afd-8742-4add-b568-05b9d17b4a4b-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.715922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715691 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.715922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.715699 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/53232afd-8742-4add-b568-05b9d17b4a4b-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:12:56.845436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.845405 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:56.850112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.849889 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:56.876637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.876602 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:56.877013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.876995 5577 cpu_manager.go:414] "containerMap: removing container" podUID="53232afd-8742-4add-b568-05b9d17b4a4b" containerName="otc-container" Oct 31 00:12:56.877093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.877015 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53232afd-8742-4add-b568-05b9d17b4a4b" containerName="otc-container" Oct 31 00:12:56.877149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.877140 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="53232afd-8742-4add-b568-05b9d17b4a4b" containerName="otc-container" Oct 31 00:12:56.881314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.881293 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:56.886029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.885991 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:12:56.886148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.886102 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:12:56.886148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.886127 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:12:56.886231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.886170 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:12:56.886231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.886186 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:12:56.886292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.886250 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:12:56.892614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:56.892577 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:57.017351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017318 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.017351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017355 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.017583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017376 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.017583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017449 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.017583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017536 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rs65\" (UniqueName: \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.017583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.017565 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118405 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118445 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118465 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118491 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118524 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rs65\" (UniqueName: \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.118787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.118551 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.119403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.119251 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.121145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.121125 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.121230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.121160 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.121522 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.121508 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.121555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.121515 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.129524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.129495 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53232afd-8742-4add-b568-05b9d17b4a4b" path="/var/lib/kubelet/pods/53232afd-8742-4add-b568-05b9d17b4a4b/volumes" Oct 31 00:12:57.131371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.131349 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rs65\" (UniqueName: \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\") pod \"data-science-collector-collector-0\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.195235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.195202 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:12:57.474009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.473934 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:12:57.477021 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:12:57.476983 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda72e0632_84bb_4975_aa90_56860480e8c3.slice/crio-e5eb88f1886eddd181005e6583ce8d639b1229441c1e7d25ffa4772796e3b9e9 WatchSource:0}: Error finding container e5eb88f1886eddd181005e6583ce8d639b1229441c1e7d25ffa4772796e3b9e9: Status 404 returned error can't find the container with id e5eb88f1886eddd181005e6583ce8d639b1229441c1e7d25ffa4772796e3b9e9 Oct 31 00:12:57.479210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.479178 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:57.479302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.479241 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:12:57.530568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:57.530542 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a72e0632-84bb-4975-aa90-56860480e8c3","Type":"ContainerStarted","Data":"e5eb88f1886eddd181005e6583ce8d639b1229441c1e7d25ffa4772796e3b9e9"} Oct 31 00:12:58.536335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:58.536244 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a72e0632-84bb-4975-aa90-56860480e8c3","Type":"ContainerStarted","Data":"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138"} Oct 31 00:12:58.557787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:12:58.557728 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.5576912800000002 podStartE2EDuration="2.55769128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:58.556474542 +0000 UTC m=+817.548134128" watchObservedRunningTime="2026-10-31 00:12:58.55769128 +0000 UTC m=+817.549350855" Oct 31 00:12:59.292769 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:12:59.292729 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:13:05.900043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:05.900009 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.099444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:06.099375 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.518430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:06.518395 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:06.671546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:06.671510 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:08.280256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:08.280213 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:08.282594 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:08.280440 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a72e0632-84bb-4975-aa90-56860480e8c3" containerName="otc-container" containerID="cri-o://120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138" gracePeriod=30 Oct 31 00:13:09.524389 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.524365 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:09.592058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.591970 5577 generic.go:356] "Generic (PLEG): container finished" podID="a72e0632-84bb-4975-aa90-56860480e8c3" containerID="120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138" exitCode=0 Oct 31 00:13:09.592058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.592019 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:09.592284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.592024 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a72e0632-84bb-4975-aa90-56860480e8c3","Type":"ContainerDied","Data":"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138"} Oct 31 00:13:09.592284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.592144 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a72e0632-84bb-4975-aa90-56860480e8c3","Type":"ContainerDied","Data":"e5eb88f1886eddd181005e6583ce8d639b1229441c1e7d25ffa4772796e3b9e9"} Oct 31 00:13:09.592284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.592170 5577 scope.go:122] "RemoveContainer" containerID="120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138" Oct 31 00:13:09.600280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.600264 5577 scope.go:122] "RemoveContainer" containerID="120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138" Oct 31 00:13:09.600542 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:09.600523 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138\": container with ID starting with 120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138 not found: ID does not exist" containerID="120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138" Oct 31 00:13:09.600605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.600551 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138"} err="failed to get container status \"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138\": rpc error: code = NotFound desc = could not find container \"120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138\": container with ID starting with 120d5e64fa51e4b8527ea6ef991a26a268ee0c6fdc68657f929d01d6952bc138 not found: ID does not exist" Oct 31 00:13:09.629503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629468 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629509 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629535 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629566 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629587 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629635 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\" (UniqueName: \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\") pod \"a72e0632-84bb-4975-aa90-56860480e8c3\" (UID: \"a72e0632-84bb-4975-aa90-56860480e8c3\") " Oct 31 00:13:09.629956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.629923 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:09.631880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.631849 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "kube-api-access-5rs65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:09.632362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.632339 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:09.632460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.632369 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:09.632460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.632433 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:09.632460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.632445 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs" pod "a72e0632-84bb-4975-aa90-56860480e8c3" (UID: "a72e0632-84bb-4975-aa90-56860480e8c3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:09.731204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731156 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rs65\" (UniqueName: \"kubernetes.io/projected/a72e0632-84bb-4975-aa90-56860480e8c3-kube-api-access-5rs65\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.731204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731191 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a72e0632-84bb-4975-aa90-56860480e8c3-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.731204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731202 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.731204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731213 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.731477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731222 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.731477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.731230 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a72e0632-84bb-4975-aa90-56860480e8c3-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:09.914949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.914916 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:09.921182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.921158 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:09.947458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.947432 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:09.947800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.947787 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a72e0632-84bb-4975-aa90-56860480e8c3" containerName="otc-container" Oct 31 00:13:09.947852 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.947801 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a72e0632-84bb-4975-aa90-56860480e8c3" containerName="otc-container" Oct 31 00:13:09.947887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.947864 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a72e0632-84bb-4975-aa90-56860480e8c3" containerName="otc-container" Oct 31 00:13:09.951992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.951976 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:09.956046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.956016 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:09.956186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.956068 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:09.956186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.956125 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:09.956270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.956245 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:09.958917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.958899 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\"" Oct 31 00:13:09.962815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:09.962792 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:10.033439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033412 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.033439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033442 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcf48\" (UniqueName: \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.033667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033467 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.033667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033496 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.033774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033665 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.033774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.033726 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.056196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.056166 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:10.134377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134344 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.134377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134392 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.135518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134428 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.135518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134442 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.135518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134468 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.135518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.134484 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcf48\" (UniqueName: \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.135518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.135424 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.137215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.137189 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.137309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.137193 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.137309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.137278 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.137309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.137298 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.147375 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.147349 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcf48\" (UniqueName: \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\") pod \"data-science-collector-collector-0\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.264545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.264509 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:10.545322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.545242 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:10.549085 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:13:10.549058 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943cff17_1272_434e_9596_0ca8919d6594.slice/crio-6399ed318cd7f72178d8dcae12ba92084b4c043d40310bfaf5193f3e89a8d7b9 WatchSource:0}: Error finding container 6399ed318cd7f72178d8dcae12ba92084b4c043d40310bfaf5193f3e89a8d7b9: Status 404 returned error can't find the container with id 6399ed318cd7f72178d8dcae12ba92084b4c043d40310bfaf5193f3e89a8d7b9 Oct 31 00:13:10.551170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.551141 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:10.551234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.551201 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:10.598961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:10.598932 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"943cff17-1272-434e-9596-0ca8919d6594","Type":"ContainerStarted","Data":"6399ed318cd7f72178d8dcae12ba92084b4c043d40310bfaf5193f3e89a8d7b9"} Oct 31 00:13:11.130837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:11.130795 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a72e0632-84bb-4975-aa90-56860480e8c3" path="/var/lib/kubelet/pods/a72e0632-84bb-4975-aa90-56860480e8c3/volumes" Oct 31 00:13:11.605432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:11.605396 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"943cff17-1272-434e-9596-0ca8919d6594","Type":"ContainerStarted","Data":"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0"} Oct 31 00:13:11.626498 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:11.626450 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.6264352 podStartE2EDuration="2.6264352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.625224829 +0000 UTC m=+830.616884413" watchObservedRunningTime="2026-10-31 00:13:11.6264352 +0000 UTC m=+830.618094780" Oct 31 00:13:18.252452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:18.252413 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.473831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:18.473791 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.845701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:18.845660 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:18.971897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:18.971857 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:21.354370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.354339 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:21.354771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.354552 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="943cff17-1272-434e-9596-0ca8919d6594" containerName="otc-container" containerID="cri-o://dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0" gracePeriod=30 Oct 31 00:13:21.601841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.601808 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.631906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.631860 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.631926 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\" (UniqueName: \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.631974 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\" (UniqueName: \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.632005 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.632042 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.632100 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\") pod \"943cff17-1272-434e-9596-0ca8919d6594\" (UID: \"943cff17-1272-434e-9596-0ca8919d6594\") " Oct 31 00:13:21.632360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.632300 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:21.632745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.632703 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/943cff17-1272-434e-9596-0ca8919d6594-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.635137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.635076 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:21.635240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.635191 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:21.635388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.635364 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:21.635388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.635378 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:21.636883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.636859 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48" pod "943cff17-1272-434e-9596-0ca8919d6594" (UID: "943cff17-1272-434e-9596-0ca8919d6594"). InnerVolumeSpecName "kube-api-access-pcf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:21.654823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.654792 5577 generic.go:356] "Generic (PLEG): container finished" podID="943cff17-1272-434e-9596-0ca8919d6594" containerID="dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0" exitCode=0 Oct 31 00:13:21.654996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.654847 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.654996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.654878 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"943cff17-1272-434e-9596-0ca8919d6594","Type":"ContainerDied","Data":"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0"} Oct 31 00:13:21.654996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.654942 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"943cff17-1272-434e-9596-0ca8919d6594","Type":"ContainerDied","Data":"6399ed318cd7f72178d8dcae12ba92084b4c043d40310bfaf5193f3e89a8d7b9"} Oct 31 00:13:21.654996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.654970 5577 scope.go:122] "RemoveContainer" containerID="dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0" Oct 31 00:13:21.663756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.663733 5577 scope.go:122] "RemoveContainer" containerID="dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0" Oct 31 00:13:21.664068 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:21.664052 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0\": container with ID starting with dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0 not found: ID does not exist" containerID="dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0" Oct 31 00:13:21.664161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.664077 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0"} err="failed to get container status \"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0\": rpc error: code = NotFound desc = could not find container \"dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0\": container with ID starting with dc6f420d2092495979c03b01f0656458d1b35ede58728fe73f1c30453e50fac0 not found: ID does not exist" Oct 31 00:13:21.676736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.676695 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:21.682089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.682065 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:21.708469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.708434 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:21.708963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.708947 5577 cpu_manager.go:414] "containerMap: removing container" podUID="943cff17-1272-434e-9596-0ca8919d6594" containerName="otc-container" Oct 31 00:13:21.709006 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.708965 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943cff17-1272-434e-9596-0ca8919d6594" containerName="otc-container" Oct 31 00:13:21.709063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.709052 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="943cff17-1272-434e-9596-0ca8919d6594" containerName="otc-container" Oct 31 00:13:21.713600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.713584 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.723348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723071 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:21.723348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723082 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:21.723348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723144 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:21.723348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723159 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:21.723348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723144 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:21.723957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.723939 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:21.724048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.724033 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:21.733245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.733226 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcf48\" (UniqueName: \"kubernetes.io/projected/943cff17-1272-434e-9596-0ca8919d6594-kube-api-access-pcf48\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.733245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.733245 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.733372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.733257 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.733372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.733267 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.733372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.733277 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/943cff17-1272-434e-9596-0ca8919d6594-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:21.834344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834265 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.834344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834300 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.834344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834322 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.834344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834337 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.834571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834447 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.834571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.834508 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dgz\" (UniqueName: \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.935788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.935746 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dgz\" (UniqueName: \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.935788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.935792 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.936035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.935817 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.936035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.935845 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.936035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.935867 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.936035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.936003 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.936862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.936833 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.938518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.938491 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.938617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.938499 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.938617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.938536 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.938688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.938665 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:21.948806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:21.948782 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dgz\" (UniqueName: \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\") pod \"data-science-collector-collector-0\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:22.026792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.026758 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:22.300643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.300361 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:22.302955 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:13:22.302920 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f573d9a_a9f4_41c0_b0a4_826810e3ab0c.slice/crio-ceff1beb3c65bab34199ff5a0ae8e0a499ad3dfebc163da2335c3046f031c281 WatchSource:0}: Error finding container ceff1beb3c65bab34199ff5a0ae8e0a499ad3dfebc163da2335c3046f031c281: Status 404 returned error can't find the container with id ceff1beb3c65bab34199ff5a0ae8e0a499ad3dfebc163da2335c3046f031c281 Oct 31 00:13:22.305352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.305321 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:22.305452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.305387 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:22.663138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.663053 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c","Type":"ContainerStarted","Data":"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b"} Oct 31 00:13:22.663138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.663115 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c","Type":"ContainerStarted","Data":"ceff1beb3c65bab34199ff5a0ae8e0a499ad3dfebc163da2335c3046f031c281"} Oct 31 00:13:22.683322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:22.683258 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.6832400220000001 podStartE2EDuration="1.683240022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:22.682932407 +0000 UTC m=+841.674591992" watchObservedRunningTime="2026-10-31 00:13:22.683240022 +0000 UTC m=+841.674899603" Oct 31 00:13:23.129891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:23.129859 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943cff17-1272-434e-9596-0ca8919d6594" path="/var/lib/kubelet/pods/943cff17-1272-434e-9596-0ca8919d6594/volumes" Oct 31 00:13:25.256304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:25.256246 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:29.292484 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:29.292445 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:13:30.317855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:30.317821 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:30.450872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:30.450819 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:30.989748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:30.987786 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:31.115865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:31.115832 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.016296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:32.016262 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:32.413860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:32.413765 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:33.421433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:33.421402 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:33.421953 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:33.421636 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" containerName="otc-container" containerID="cri-o://9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b" gracePeriod=30 Oct 31 00:13:34.662567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.662543 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:34.717527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.717499 5577 generic.go:356] "Generic (PLEG): container finished" podID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" containerID="9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b" exitCode=0 Oct 31 00:13:34.717657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.717546 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:34.717657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.717596 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c","Type":"ContainerDied","Data":"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b"} Oct 31 00:13:34.717657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.717647 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c","Type":"ContainerDied","Data":"ceff1beb3c65bab34199ff5a0ae8e0a499ad3dfebc163da2335c3046f031c281"} Oct 31 00:13:34.717797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.717671 5577 scope.go:122] "RemoveContainer" containerID="9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b" Oct 31 00:13:34.725959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.725942 5577 scope.go:122] "RemoveContainer" containerID="9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b" Oct 31 00:13:34.726216 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:34.726198 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b\": container with ID starting with 9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b not found: ID does not exist" containerID="9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b" Oct 31 00:13:34.726257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.726227 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b"} err="failed to get container status \"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b\": rpc error: code = NotFound desc = could not find container \"9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b\": container with ID starting with 9f15c4855a0583c5dc78a81f57c8fd45974cb74e9cd500cc3be26ca08a97664b not found: ID does not exist" Oct 31 00:13:34.741502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741483 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\" (UniqueName: \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741513 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741551 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741577 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741721 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741623 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741721 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741674 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\") pod \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\" (UID: \"4f573d9a-a9f4-41c0-b0a4-826810e3ab0c\") " Oct 31 00:13:34.741952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.741922 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:34.743804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.743778 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:34.743889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.743845 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "kube-api-access-j2dgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:34.743947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.743886 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:34.744335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.744307 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:34.744416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.744335 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs" pod "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" (UID: "4f573d9a-a9f4-41c0-b0a4-826810e3ab0c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842840 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842867 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842878 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842889 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842898 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dgz\" (UniqueName: \"kubernetes.io/projected/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-kube-api-access-j2dgz\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:34.842910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:34.842907 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:35.038971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.038938 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:35.045298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.045276 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:35.072828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.072806 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:35.073180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.073167 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" containerName="otc-container" Oct 31 00:13:35.073221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.073183 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" containerName="otc-container" Oct 31 00:13:35.073265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.073256 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" containerName="otc-container" Oct 31 00:13:35.077513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.077499 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.081581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081558 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:35.081581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081558 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:35.081774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081601 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:35.081774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081606 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:35.081774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081654 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:35.081959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.081946 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\"" Oct 31 00:13:35.086985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.086967 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:35.129679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.129606 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f573d9a-a9f4-41c0-b0a4-826810e3ab0c" path="/var/lib/kubelet/pods/4f573d9a-a9f4-41c0-b0a4-826810e3ab0c/volumes" Oct 31 00:13:35.145388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145365 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.145503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145390 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.145503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145409 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.145503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145426 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.145503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145486 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.145647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.145603 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnrl\" (UniqueName: \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246425 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnrl\" (UniqueName: \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246484 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246508 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246533 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246554 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.246885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.246744 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.247287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.247263 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.249239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.249216 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.249355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.249258 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.249355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.249330 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.249448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.249375 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.258763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.258746 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnrl\" (UniqueName: \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\") pod \"data-science-collector-collector-0\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.393627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.393549 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:35.669373 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.668955 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:35.671779 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:13:35.671750 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cbf0cf6_191f_4686_b0af_d777bfdaae75.slice/crio-f589489dc9d7bd92eb9cc26432799ff54dd36a91c320a221584ef3c7c4ec8104 WatchSource:0}: Error finding container f589489dc9d7bd92eb9cc26432799ff54dd36a91c320a221584ef3c7c4ec8104: Status 404 returned error can't find the container with id f589489dc9d7bd92eb9cc26432799ff54dd36a91c320a221584ef3c7c4ec8104 Oct 31 00:13:35.674630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.674598 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:35.674788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.674656 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:35.722902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:35.722872 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbf0cf6-191f-4686-b0af-d777bfdaae75","Type":"ContainerStarted","Data":"f589489dc9d7bd92eb9cc26432799ff54dd36a91c320a221584ef3c7c4ec8104"} Oct 31 00:13:36.730589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:36.730546 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbf0cf6-191f-4686-b0af-d777bfdaae75","Type":"ContainerStarted","Data":"0f56f6117ac37e75cc4ec60aa2216e07f45d396c7d4028ba5c9a49a46f9c1dba"} Oct 31 00:13:36.751542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:36.751491 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.751474118 podStartE2EDuration="1.751474118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:36.750517908 +0000 UTC m=+855.742177503" watchObservedRunningTime="2026-10-31 00:13:36.751474118 +0000 UTC m=+855.743133734" Oct 31 00:13:38.233773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:38.233738 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:38.363751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:38.363694 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.727837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:42.727798 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:42.871824 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:42.871740 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:43.338419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:43.338338 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:43.473129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:43.473073 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:43.622453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:43.622365 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:47.500075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:47.500044 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:47.500567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:47.500283 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" containerName="otc-container" containerID="cri-o://0f56f6117ac37e75cc4ec60aa2216e07f45d396c7d4028ba5c9a49a46f9c1dba" gracePeriod=30 Oct 31 00:13:49.790857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.790828 5577 generic.go:356] "Generic (PLEG): container finished" podID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" containerID="0f56f6117ac37e75cc4ec60aa2216e07f45d396c7d4028ba5c9a49a46f9c1dba" exitCode=0 Oct 31 00:13:49.791247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.790903 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbf0cf6-191f-4686-b0af-d777bfdaae75","Type":"ContainerDied","Data":"0f56f6117ac37e75cc4ec60aa2216e07f45d396c7d4028ba5c9a49a46f9c1dba"} Oct 31 00:13:49.851922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.851889 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:49.974860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.974758 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.974860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.974840 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\" (UniqueName: \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.975100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.974932 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.975100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.974979 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.975100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.975005 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.975100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.975031 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\") pod \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\" (UID: \"5cbf0cf6-191f-4686-b0af-d777bfdaae75\") " Oct 31 00:13:49.975343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.975313 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:49.977332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.977299 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:49.977679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.977645 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:49.977679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.977668 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "kube-api-access-7gnrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:49.977979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.977965 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:49.978034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:49.977994 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs" pod "5cbf0cf6-191f-4686-b0af-d777bfdaae75" (UID: "5cbf0cf6-191f-4686-b0af-d777bfdaae75"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:50.075985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.075948 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gnrl\" (UniqueName: \"kubernetes.io/projected/5cbf0cf6-191f-4686-b0af-d777bfdaae75-kube-api-access-7gnrl\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.075985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.075980 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.075985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.075989 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.075985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.075998 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.076270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.076007 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.076270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.076015 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5cbf0cf6-191f-4686-b0af-d777bfdaae75-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:50.798753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.798693 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5cbf0cf6-191f-4686-b0af-d777bfdaae75","Type":"ContainerDied","Data":"f589489dc9d7bd92eb9cc26432799ff54dd36a91c320a221584ef3c7c4ec8104"} Oct 31 00:13:50.799155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.798789 5577 scope.go:122] "RemoveContainer" containerID="0f56f6117ac37e75cc4ec60aa2216e07f45d396c7d4028ba5c9a49a46f9c1dba" Oct 31 00:13:50.799155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.798729 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.820531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.820496 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:50.825560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.825512 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:50.851684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.851646 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:50.852040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.852026 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" containerName="otc-container" Oct 31 00:13:50.852089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.852041 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" containerName="otc-container" Oct 31 00:13:50.852129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.852092 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" containerName="otc-container" Oct 31 00:13:50.856529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.856513 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.860626 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860602 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:50.860783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860761 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:50.860783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860778 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:50.860878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860787 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:50.860878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860762 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:50.860878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.860817 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:50.865697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.865675 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:50.983957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.983925 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfnsv\" (UniqueName: \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.984134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.983972 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.984134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.984072 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.984223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.984183 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.984265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.984221 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:50.984265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:50.984245 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.084934 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.084979 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.085005 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.085035 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfnsv\" (UniqueName: \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.085079 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.085108 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.085808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.085753 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.087940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.087910 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.088045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.087921 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.088242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.088220 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.088289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.088220 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.097506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.097483 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfnsv\" (UniqueName: \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\") pod \"data-science-collector-collector-0\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.129799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.129766 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cbf0cf6-191f-4686-b0af-d777bfdaae75" path="/var/lib/kubelet/pods/5cbf0cf6-191f-4686-b0af-d777bfdaae75/volumes" Oct 31 00:13:51.170937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.170902 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:51.448155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.448041 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:51.451068 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:13:51.451031 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ae1ae72_47d0_4b2d_9c4d_4df517758f40.slice/crio-2d27118ec18500625e9862f176d481e42db7209d8ee53fad092d55ef12b180c3 WatchSource:0}: Error finding container 2d27118ec18500625e9862f176d481e42db7209d8ee53fad092d55ef12b180c3: Status 404 returned error can't find the container with id 2d27118ec18500625e9862f176d481e42db7209d8ee53fad092d55ef12b180c3 Oct 31 00:13:51.453573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.453540 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:51.453676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.453608 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:51.805610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.805576 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ae1ae72-47d0-4b2d-9c4d-4df517758f40","Type":"ContainerStarted","Data":"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b"} Oct 31 00:13:51.806050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.805630 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ae1ae72-47d0-4b2d-9c4d-4df517758f40","Type":"ContainerStarted","Data":"2d27118ec18500625e9862f176d481e42db7209d8ee53fad092d55ef12b180c3"} Oct 31 00:13:51.827405 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:51.827357 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.82734463 podStartE2EDuration="1.82734463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:51.825340497 +0000 UTC m=+870.817000082" watchObservedRunningTime="2026-10-31 00:13:51.82734463 +0000 UTC m=+870.819004215" Oct 31 00:13:54.979985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:54.979946 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.181036 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:55.180990 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.577564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:55.577428 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:55.707466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:55.707421 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:13:58.558041 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.558010 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:58.558471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.558229 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" containerName="otc-container" containerID="cri-o://a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b" gracePeriod=30 Oct 31 00:13:58.804306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.804277 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:58.841860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.841782 5577 generic.go:356] "Generic (PLEG): container finished" podID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" containerID="a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b" exitCode=0 Oct 31 00:13:58.841860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.841827 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:58.841860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.841844 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ae1ae72-47d0-4b2d-9c4d-4df517758f40","Type":"ContainerDied","Data":"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b"} Oct 31 00:13:58.842105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.841886 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5ae1ae72-47d0-4b2d-9c4d-4df517758f40","Type":"ContainerDied","Data":"2d27118ec18500625e9862f176d481e42db7209d8ee53fad092d55ef12b180c3"} Oct 31 00:13:58.842105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.841907 5577 scope.go:122] "RemoveContainer" containerID="a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b" Oct 31 00:13:58.850290 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.850268 5577 scope.go:122] "RemoveContainer" containerID="a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b" Oct 31 00:13:58.850571 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:58.850555 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b\": container with ID starting with a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b not found: ID does not exist" containerID="a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b" Oct 31 00:13:58.850618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.850588 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b"} err="failed to get container status \"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b\": rpc error: code = NotFound desc = could not find container \"a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b\": container with ID starting with a54460bb574b06af3e1c8d16c120c9b4bc373ce01cc6af53080b0d72a3b2851b not found: ID does not exist" Oct 31 00:13:58.946570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946531 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.946777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946596 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\" (UniqueName: \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.946777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946626 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.946777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946685 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.946914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946794 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.946914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.946884 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\") pod \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\" (UID: \"5ae1ae72-47d0-4b2d-9c4d-4df517758f40\") " Oct 31 00:13:58.947041 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.947016 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:13:58.947161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.947145 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:58.949126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.949093 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:58.949229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.949123 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:58.949229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.949133 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:58.949385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.949369 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:13:58.949385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:58.949375 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv" pod "5ae1ae72-47d0-4b2d-9c4d-4df517758f40" (UID: "5ae1ae72-47d0-4b2d-9c4d-4df517758f40"). InnerVolumeSpecName "kube-api-access-nfnsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:13:59.048127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.048084 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfnsv\" (UniqueName: \"kubernetes.io/projected/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-kube-api-access-nfnsv\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:59.048127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.048122 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:59.048338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.048137 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:59.048338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.048149 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:59.048338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.048162 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ae1ae72-47d0-4b2d-9c4d-4df517758f40-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:13:59.157511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.157479 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:59.161927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.161893 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:59.188857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.188826 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:59.189343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.189328 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" containerName="otc-container" Oct 31 00:13:59.189384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.189347 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" containerName="otc-container" Oct 31 00:13:59.189445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.189433 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" containerName="otc-container" Oct 31 00:13:59.193996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.193976 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.201384 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.201874 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.201985 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.202081 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.202167 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.202251 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:13:59.202366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.202347 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:13:59.292193 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:13:59.292148 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:13:59.350287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350251 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.350432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350306 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.350432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350370 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.350432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350400 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.350554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350428 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvzr7\" (UniqueName: \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.350590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.350552 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.451930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.451820 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.452113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.451944 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.452113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.452017 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvzr7\" (UniqueName: \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.452113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.452097 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.452284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.452137 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.452284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.452180 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.453782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.453754 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.459616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.459591 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.459763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.459747 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.459855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.459835 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.460804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.460780 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.465350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.465327 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvzr7\" (UniqueName: \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\") pod \"data-science-collector-collector-0\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.509306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.509276 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:13:59.784887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.784851 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:13:59.788624 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:13:59.788596 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f7c8739_f6cc_44fe_9dc9_7d12380dd4fb.slice/crio-1c3ff81ad24c686123b396ab363705a163e9ab7aa0ecbe1dfac543e38f2f9f3a WatchSource:0}: Error finding container 1c3ff81ad24c686123b396ab363705a163e9ab7aa0ecbe1dfac543e38f2f9f3a: Status 404 returned error can't find the container with id 1c3ff81ad24c686123b396ab363705a163e9ab7aa0ecbe1dfac543e38f2f9f3a Oct 31 00:13:59.791292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.791260 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:59.791389 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.791321 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:13:59.847638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:13:59.847606 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb","Type":"ContainerStarted","Data":"1c3ff81ad24c686123b396ab363705a163e9ab7aa0ecbe1dfac543e38f2f9f3a"} Oct 31 00:14:00.855995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:00.855961 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb","Type":"ContainerStarted","Data":"771838025b412f7ec7e8ae06df77a2a9c1020f7bddb0e1ce070a18b73d893017"} Oct 31 00:14:00.876934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:00.876880 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.876866572 podStartE2EDuration="1.876866572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:00.875260695 +0000 UTC m=+879.866920274" watchObservedRunningTime="2026-10-31 00:14:00.876866572 +0000 UTC m=+879.868526147" Oct 31 00:14:01.130198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:01.130119 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ae1ae72-47d0-4b2d-9c4d-4df517758f40" path="/var/lib/kubelet/pods/5ae1ae72-47d0-4b2d-9c4d-4df517758f40/volumes" Oct 31 00:14:07.166567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:07.166503 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.293696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:07.293655 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.774315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:07.774280 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:07.898516 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:07.898478 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:10.624393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:10.624362 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:10.624810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:10.624535 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" containerName="otc-container" containerID="cri-o://771838025b412f7ec7e8ae06df77a2a9c1020f7bddb0e1ce070a18b73d893017" gracePeriod=30 Oct 31 00:14:12.920048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:12.920011 5577 generic.go:356] "Generic (PLEG): container finished" podID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" containerID="771838025b412f7ec7e8ae06df77a2a9c1020f7bddb0e1ce070a18b73d893017" exitCode=0 Oct 31 00:14:12.920417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:12.920077 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb","Type":"ContainerDied","Data":"771838025b412f7ec7e8ae06df77a2a9c1020f7bddb0e1ce070a18b73d893017"} Oct 31 00:14:12.968412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:12.968387 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:13.062554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062523 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.062554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062557 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.062845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062594 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.062845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062623 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.062845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062662 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.063026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.062994 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:13.065025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.064990 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:13.065025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.065003 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:13.065190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.065058 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:13.065190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.065080 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:13.163258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163227 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\" (UniqueName: \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\") pod \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\" (UID: \"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb\") " Oct 31 00:14:13.163664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163521 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.163664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163548 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.163664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163564 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.163664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163579 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.163664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.163594 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.165509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.165485 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7" pod "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" (UID: "5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb"). InnerVolumeSpecName "kube-api-access-hvzr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:13.264768 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.264735 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvzr7\" (UniqueName: \"kubernetes.io/projected/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb-kube-api-access-hvzr7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:13.926165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.926139 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:13.926642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.926139 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb","Type":"ContainerDied","Data":"1c3ff81ad24c686123b396ab363705a163e9ab7aa0ecbe1dfac543e38f2f9f3a"} Oct 31 00:14:13.926642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.926273 5577 scope.go:122] "RemoveContainer" containerID="771838025b412f7ec7e8ae06df77a2a9c1020f7bddb0e1ce070a18b73d893017" Oct 31 00:14:13.949079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.949051 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:13.952580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.952559 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:13.980819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.980796 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:13.981213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.981198 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" containerName="otc-container" Oct 31 00:14:13.981255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.981217 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" containerName="otc-container" Oct 31 00:14:13.981303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.981294 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" containerName="otc-container" Oct 31 00:14:13.985854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.985836 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:13.990098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990077 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:13.990249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990113 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:13.990328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990144 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:13.990328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990176 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:13.990328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990219 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:13.990886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.990871 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:13.994763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:13.994741 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:14.072956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.072928 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.073123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.072967 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.073123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.072996 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkq8s\" (UniqueName: \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.073215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.073179 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.073337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.073312 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.073380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.073361 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174464 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174514 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174553 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174581 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174612 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkq8s\" (UniqueName: \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.174692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.174654 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.175341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.175293 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.177550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.177521 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.177738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.177695 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.178034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.178013 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.178121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.178107 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.187809 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.187786 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkq8s\" (UniqueName: \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\") pod \"data-science-collector-collector-0\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.301803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.301776 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:14.578115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.578081 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:14.578528 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:14:14.578501 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9eb75_892b_4645_a9b4_0179a35c1774.slice/crio-f8dd368f112b16e8c4783db68f34ebe273e36b65394a177b132a79a6aeb25925 WatchSource:0}: Error finding container f8dd368f112b16e8c4783db68f34ebe273e36b65394a177b132a79a6aeb25925: Status 404 returned error can't find the container with id f8dd368f112b16e8c4783db68f34ebe273e36b65394a177b132a79a6aeb25925 Oct 31 00:14:14.580769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.580699 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:14.580833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.580802 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:14.933340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.933244 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d2d9eb75-892b-4645-a9b4-0179a35c1774","Type":"ContainerStarted","Data":"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180"} Oct 31 00:14:14.933340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.933303 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d2d9eb75-892b-4645-a9b4-0179a35c1774","Type":"ContainerStarted","Data":"f8dd368f112b16e8c4783db68f34ebe273e36b65394a177b132a79a6aeb25925"} Oct 31 00:14:14.953972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:14.953888 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.9538747509999999 podStartE2EDuration="1.953874751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:14.953226427 +0000 UTC m=+893.944886013" watchObservedRunningTime="2026-10-31 00:14:14.953874751 +0000 UTC m=+893.945534402" Oct 31 00:14:15.132555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:15.131069 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb" path="/var/lib/kubelet/pods/5f7c8739-f6cc-44fe-9dc9-7d12380dd4fb/volumes" Oct 31 00:14:19.389732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:19.389677 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:20.775503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:20.775468 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:21.219840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:21.219765 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:14:21.221501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:21.221468 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:14:25.306168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:25.306118 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:29.293959 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:14:29.293859 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:14:29.732407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:29.732332 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:29.732597 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:29.732573 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d2d9eb75-892b-4645-a9b4-0179a35c1774" containerName="otc-container" containerID="cri-o://585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180" gracePeriod=30 Oct 31 00:14:30.968547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:30.968521 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.009221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.009192 5577 generic.go:356] "Generic (PLEG): container finished" podID="d2d9eb75-892b-4645-a9b4-0179a35c1774" containerID="585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180" exitCode=0 Oct 31 00:14:31.009341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.009239 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.009341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.009229 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d2d9eb75-892b-4645-a9b4-0179a35c1774","Type":"ContainerDied","Data":"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180"} Oct 31 00:14:31.009341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.009330 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d2d9eb75-892b-4645-a9b4-0179a35c1774","Type":"ContainerDied","Data":"f8dd368f112b16e8c4783db68f34ebe273e36b65394a177b132a79a6aeb25925"} Oct 31 00:14:31.009466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.009359 5577 scope.go:122] "RemoveContainer" containerID="585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180" Oct 31 00:14:31.014545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014518 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\" (UniqueName: \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.014665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014553 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.014665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014580 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.014665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014627 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.014841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014668 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.014841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.014750 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\") pod \"d2d9eb75-892b-4645-a9b4-0179a35c1774\" (UID: \"d2d9eb75-892b-4645-a9b4-0179a35c1774\") " Oct 31 00:14:31.015029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.015000 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:31.017501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.017351 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:31.017501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.017449 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "kube-api-access-gkq8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:31.017671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.017522 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:31.017671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.017605 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:31.017837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.017820 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs" pod "d2d9eb75-892b-4645-a9b4-0179a35c1774" (UID: "d2d9eb75-892b-4645-a9b4-0179a35c1774"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:31.021471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.021450 5577 scope.go:122] "RemoveContainer" containerID="585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180" Oct 31 00:14:31.021805 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:14:31.021788 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180\": container with ID starting with 585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180 not found: ID does not exist" containerID="585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180" Oct 31 00:14:31.021872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.021813 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180"} err="failed to get container status \"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180\": rpc error: code = NotFound desc = could not find container \"585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180\": container with ID starting with 585926b41c6fb8f878c3603196cee698bb07dbd35b7a2e22caa9a4977e5ba180 not found: ID does not exist" Oct 31 00:14:31.115537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115468 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkq8s\" (UniqueName: \"kubernetes.io/projected/d2d9eb75-892b-4645-a9b4-0179a35c1774-kube-api-access-gkq8s\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.115537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115495 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2d9eb75-892b-4645-a9b4-0179a35c1774-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.115537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115508 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.115537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115521 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.115537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115534 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.115803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.115547 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2d9eb75-892b-4645-a9b4-0179a35c1774-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:31.328038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.328007 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:31.332020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.331995 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:31.357900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.357876 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:31.358234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.358222 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d2d9eb75-892b-4645-a9b4-0179a35c1774" containerName="otc-container" Oct 31 00:14:31.358270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.358236 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d9eb75-892b-4645-a9b4-0179a35c1774" containerName="otc-container" Oct 31 00:14:31.358311 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.358289 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d9eb75-892b-4645-a9b4-0179a35c1774" containerName="otc-container" Oct 31 00:14:31.363057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.363039 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.367171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367103 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:31.367171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367156 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:31.367838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367784 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:31.367838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367804 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:31.367838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367807 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:31.367838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.367820 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:31.371947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.371924 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:31.418152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418124 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.418276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418189 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.418276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418259 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.418349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418320 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.418393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418380 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.418464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.418448 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdrxm\" (UniqueName: \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.438930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.438905 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.519424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519398 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.519572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519428 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.519572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519447 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.519572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519481 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.519572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519499 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdrxm\" (UniqueName: \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.519572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.519561 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.520491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.520467 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.522148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.522125 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.522256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.522239 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.522302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.522261 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.522340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.522301 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.531522 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.531501 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdrxm\" (UniqueName: \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\") pod \"data-science-collector-collector-0\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.573196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.573162 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:31.677653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.677586 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:31.959434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.959356 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:31.962844 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:14:31.962811 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e18aa19_3830_4160_bf38_6e5d7e35e1af.slice/crio-19419f5cf44272b2e65e2ad395df157dc07aa7153dbf1d6e27d3f7d8bf9032c3 WatchSource:0}: Error finding container 19419f5cf44272b2e65e2ad395df157dc07aa7153dbf1d6e27d3f7d8bf9032c3: Status 404 returned error can't find the container with id 19419f5cf44272b2e65e2ad395df157dc07aa7153dbf1d6e27d3f7d8bf9032c3 Oct 31 00:14:31.965203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.965165 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:31.965293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:31.965246 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:32.017176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:32.017150 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e18aa19-3830-4160-bf38-6e5d7e35e1af","Type":"ContainerStarted","Data":"19419f5cf44272b2e65e2ad395df157dc07aa7153dbf1d6e27d3f7d8bf9032c3"} Oct 31 00:14:32.036069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:32.035987 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:32.162569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:32.162529 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.023572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.023543 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e18aa19-3830-4160-bf38-6e5d7e35e1af","Type":"ContainerStarted","Data":"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484"} Oct 31 00:14:33.044112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.044065 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.04405149 podStartE2EDuration="2.04405149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:33.042629444 +0000 UTC m=+912.034289030" watchObservedRunningTime="2026-10-31 00:14:33.04405149 +0000 UTC m=+912.035711068" Oct 31 00:14:33.129542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.129499 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d9eb75-892b-4645-a9b4-0179a35c1774" path="/var/lib/kubelet/pods/d2d9eb75-892b-4645-a9b4-0179a35c1774/volumes" Oct 31 00:14:33.555063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.555028 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.755827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.755795 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:33.763338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:33.763311 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:34.461920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:34.461883 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:35.031327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.031290 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" containerName="otc-container" containerID="cri-o://ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484" gracePeriod=30 Oct 31 00:14:35.279500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.279469 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:35.353370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353308 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353354 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353386 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353423 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353479 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353530 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\" (UniqueName: \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\") pod \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\" (UID: \"5e18aa19-3830-4160-bf38-6e5d7e35e1af\") " Oct 31 00:14:35.353981 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.353941 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:35.355803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.355775 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "kube-api-access-sdrxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:35.356175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.356158 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:35.356299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.356277 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:35.356359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.356293 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:35.356359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.356304 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs" pod "5e18aa19-3830-4160-bf38-6e5d7e35e1af" (UID: "5e18aa19-3830-4160-bf38-6e5d7e35e1af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:35.454464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454435 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.454464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454460 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5e18aa19-3830-4160-bf38-6e5d7e35e1af-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.454653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454471 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdrxm\" (UniqueName: \"kubernetes.io/projected/5e18aa19-3830-4160-bf38-6e5d7e35e1af-kube-api-access-sdrxm\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.454653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454482 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.454653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454490 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.454653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.454500 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5e18aa19-3830-4160-bf38-6e5d7e35e1af-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:35.957176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:35.957132 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.037702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.037670 5577 generic.go:356] "Generic (PLEG): container finished" podID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" containerID="ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484" exitCode=0 Oct 31 00:14:36.037896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.037740 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e18aa19-3830-4160-bf38-6e5d7e35e1af","Type":"ContainerDied","Data":"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484"} Oct 31 00:14:36.037896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.037803 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.037896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.037837 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5e18aa19-3830-4160-bf38-6e5d7e35e1af","Type":"ContainerDied","Data":"19419f5cf44272b2e65e2ad395df157dc07aa7153dbf1d6e27d3f7d8bf9032c3"} Oct 31 00:14:36.037896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.037870 5577 scope.go:122] "RemoveContainer" containerID="ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484" Oct 31 00:14:36.047565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.047545 5577 scope.go:122] "RemoveContainer" containerID="ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484" Oct 31 00:14:36.047859 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:14:36.047839 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484\": container with ID starting with ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484 not found: ID does not exist" containerID="ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484" Oct 31 00:14:36.047926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.047872 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484"} err="failed to get container status \"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484\": rpc error: code = NotFound desc = could not find container \"ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484\": container with ID starting with ecbbde2ffbe2496fa4b50832f0fbd504d64182f162dc9b3471919478c8f3d484 not found: ID does not exist" Oct 31 00:14:36.060188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.060165 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:36.064422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.064403 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:36.091750 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.091724 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:36.092101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.092087 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" containerName="otc-container" Oct 31 00:14:36.092153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.092104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" containerName="otc-container" Oct 31 00:14:36.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.092173 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" containerName="otc-container" Oct 31 00:14:36.096647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.096630 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.098379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.098353 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:36.100617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100598 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:36.100776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100757 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:36.100830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100759 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:36.100830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100806 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:36.100902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100872 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:36.100938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.100917 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:36.105249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.105227 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:36.160373 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160348 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.160541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160378 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.160541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160403 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.160620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160534 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r592r\" (UniqueName: \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.160620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160600 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.160735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.160695 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.261732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261681 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r592r\" (UniqueName: \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.261931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261743 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.261931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261761 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.261931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261791 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.261931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261810 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.262115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.261948 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.262894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.262866 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.264544 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.264514 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.264658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.264549 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.264658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.264523 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.264756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.264667 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.273590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.273567 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r592r\" (UniqueName: \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\") pod \"data-science-collector-collector-0\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.410355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.410324 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:36.683721 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.683619 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:36.686848 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:14:36.686814 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434c351f_7f2e_4d8a_a6ba_78250bc9e2a1.slice/crio-290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2 WatchSource:0}: Error finding container 290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2: Status 404 returned error can't find the container with id 290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2 Oct 31 00:14:36.689518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.689485 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:36.689617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:36.689569 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:37.012977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.012935 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:37.043530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.043482 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1","Type":"ContainerStarted","Data":"66426cc63ecc5d62f9a2b8282c510936ed1a04b2fe9b1b8a1f04495a0ff94f22"} Oct 31 00:14:37.043740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.043614 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1","Type":"ContainerStarted","Data":"290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2"} Oct 31 00:14:37.067103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.067049 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.0670364860000001 podStartE2EDuration="1.067036486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:37.065169881 +0000 UTC m=+916.056829477" watchObservedRunningTime="2026-10-31 00:14:37.067036486 +0000 UTC m=+916.058696066" Oct 31 00:14:37.130550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.130516 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e18aa19-3830-4160-bf38-6e5d7e35e1af" path="/var/lib/kubelet/pods/5e18aa19-3830-4160-bf38-6e5d7e35e1af/volumes" Oct 31 00:14:37.157570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:37.157535 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.517914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:39.517873 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:39.720562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:39.720525 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.483460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:40.483423 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:40.617360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:40.617319 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:45.710020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:45.709979 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:45.841263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:45.841229 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:49.720988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:49.720955 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:49.846730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:49.846664 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:50.343938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:50.343898 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:50.473528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:50.473486 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.818607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:52.818572 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:52.949332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:52.949297 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:14:53.882886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:53.882845 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:53.883340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:53.883077 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" containerName="otc-container" containerID="cri-o://66426cc63ecc5d62f9a2b8282c510936ed1a04b2fe9b1b8a1f04495a0ff94f22" gracePeriod=30 Oct 31 00:14:54.120924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.120893 5577 generic.go:356] "Generic (PLEG): container finished" podID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" containerID="66426cc63ecc5d62f9a2b8282c510936ed1a04b2fe9b1b8a1f04495a0ff94f22" exitCode=0 Oct 31 00:14:54.121058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.120956 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1","Type":"ContainerDied","Data":"66426cc63ecc5d62f9a2b8282c510936ed1a04b2fe9b1b8a1f04495a0ff94f22"} Oct 31 00:14:54.121058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.121005 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1","Type":"ContainerDied","Data":"290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2"} Oct 31 00:14:54.121058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.121030 5577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="290b295f6f9df2100256f4b3c0c656ebb588b02ba13b143c8265e209d604a4b2" Oct 31 00:14:54.130754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.130732 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:54.215627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215546 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.215627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215616 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.215888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215643 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.215888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215680 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.215888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215731 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.215888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.215786 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\" (UniqueName: \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\") pod \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\" (UID: \"434c351f-7f2e-4d8a-a6ba-78250bc9e2a1\") " Oct 31 00:14:54.216174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.216143 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:14:54.217937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.217914 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:54.218350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.218326 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:54.218448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.218399 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:54.218502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.218459 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:14:54.218543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.218527 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r" pod "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" (UID: "434c351f-7f2e-4d8a-a6ba-78250bc9e2a1"). InnerVolumeSpecName "kube-api-access-r592r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:14:54.317100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317070 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.317100 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317101 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.317267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317118 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r592r\" (UniqueName: \"kubernetes.io/projected/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-kube-api-access-r592r\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.317267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317131 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.317267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317140 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:54.317267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:54.317164 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:14:55.125109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.125075 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.147452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.147425 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:55.152033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.152012 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:55.178240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.178219 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:55.178536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.178525 5577 cpu_manager.go:414] "containerMap: removing container" podUID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" containerName="otc-container" Oct 31 00:14:55.178582 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.178538 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" containerName="otc-container" Oct 31 00:14:55.178619 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.178603 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" containerName="otc-container" Oct 31 00:14:55.181692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.181679 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.185480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.185461 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:14:55.185832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.185811 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:14:55.186153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.186136 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:14:55.186267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.186159 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:14:55.186267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.186173 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:14:55.186267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.186191 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:14:55.191538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.191518 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:55.324808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.324779 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.324992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.324816 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.324992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.324839 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.324992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.324901 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.324992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.324974 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.325127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.325001 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js27f\" (UniqueName: \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425570 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425605 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425641 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425662 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js27f\" (UniqueName: \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425771 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.425915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.425801 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.426702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.426674 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.428245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.428220 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.428378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.428362 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.428604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.428419 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.428604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.428497 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.437989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.437965 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js27f\" (UniqueName: \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\") pod \"data-science-collector-collector-0\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.494595 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.494574 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:14:55.769473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.769354 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:14:55.772244 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:14:55.772218 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b51430_a30a_4b30_a69f_867792d4ef14.slice/crio-5ef2230e6777c9653dffbbd80d5bab19739ef138435f08bb2ce9958d002d7a57 WatchSource:0}: Error finding container 5ef2230e6777c9653dffbbd80d5bab19739ef138435f08bb2ce9958d002d7a57: Status 404 returned error can't find the container with id 5ef2230e6777c9653dffbbd80d5bab19739ef138435f08bb2ce9958d002d7a57 Oct 31 00:14:55.774309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.774282 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:55.774375 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:55.774339 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:14:56.131874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:56.131786 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a8b51430-a30a-4b30-a69f-867792d4ef14","Type":"ContainerStarted","Data":"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5"} Oct 31 00:14:56.131874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:56.131843 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a8b51430-a30a-4b30-a69f-867792d4ef14","Type":"ContainerStarted","Data":"5ef2230e6777c9653dffbbd80d5bab19739ef138435f08bb2ce9958d002d7a57"} Oct 31 00:14:56.151674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:56.151626 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.151613781 podStartE2EDuration="1.151613781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:56.150474188 +0000 UTC m=+935.142133782" watchObservedRunningTime="2026-10-31 00:14:56.151613781 +0000 UTC m=+935.143273362" Oct 31 00:14:57.129551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:57.129516 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="434c351f-7f2e-4d8a-a6ba-78250bc9e2a1" path="/var/lib/kubelet/pods/434c351f-7f2e-4d8a-a6ba-78250bc9e2a1/volumes" Oct 31 00:14:59.146720 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:59.146682 5577 generic.go:356] "Generic (PLEG): container finished" podID="c2d7942b-59c9-4d34-bf75-4156c16e5bbd" containerID="fefbab087abcbb36cce9c5ca2a51e8677c1552fbc5bd7daa1b06c50e74d894b5" exitCode=137 Oct 31 00:14:59.147134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:59.146740 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerDied","Data":"fefbab087abcbb36cce9c5ca2a51e8677c1552fbc5bd7daa1b06c50e74d894b5"} Oct 31 00:14:59.147134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:59.147105 5577 scope.go:122] "RemoveContainer" containerID="fefbab087abcbb36cce9c5ca2a51e8677c1552fbc5bd7daa1b06c50e74d894b5" Oct 31 00:14:59.147449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:59.147436 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:14:59.290351 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:14:59.290321 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:14:59.884638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:14:59.884606 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:15:00.155791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:00.155762 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerStarted","Data":"fc446824bbd388f4c8c612a5340001dcc3d5b6840796a2efc40f7973a0c77769"} Oct 31 00:15:00.156182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:00.155884 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:15:11.162186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:11.162154 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:15:23.868987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:23.868952 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:23.995809 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:23.995757 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:25.355903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:15:25.355868 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:15:29.292047 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:15:29.292012 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:15:59.291733 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:15:59.291622 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:16:14.898092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:14.898052 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:15.023459 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:15.023427 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:18.014305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:18.014255 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:18.143262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:18.143224 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.051110 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:21.051078 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:21.177314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:21.177272 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:23.015079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:23.015031 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.105674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:25.105634 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.230918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:25.230881 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:25.402666 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:25.402580 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:28.913492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:28.913460 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.040301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:29.040267 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.207231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:29.207145 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:29.293398 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:16:29.293365 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:16:29.336555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:29.336530 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:34.631784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:34.631741 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.165797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.165760 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.298496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.298457 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.814397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.814342 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.839522 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.839488 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:16:47.842929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.842907 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:47.846681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.846650 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:16:47.846841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.846826 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:16:47.847395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.847370 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-zpbg7\"" Oct 31 00:16:47.853880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.853855 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:16:47.940813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.940768 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:16:47.951800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.951774 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:47.951931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.951804 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh7cf\" (UniqueName: \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:47.951973 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:47.951929 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.052795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.052757 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.053040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.052845 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.053040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.052873 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh7cf\" (UniqueName: \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.053453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.053432 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.055538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.055516 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.065410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.065350 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh7cf\" (UniqueName: \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\") pod \"data-science-collector-targetallocator-db564bcf8-vs7q9\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.154325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.154294 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.433204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.433021 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:16:48.436105 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:16:48.436080 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5b0197f_acd3_47bd_bb97_1f6c0760b007.slice/crio-d1aebb17c25dbae7c782045b16f4af85011feebc266ca167953b82684a10bf4b WatchSource:0}: Error finding container d1aebb17c25dbae7c782045b16f4af85011feebc266ca167953b82684a10bf4b: Status 404 returned error can't find the container with id d1aebb17c25dbae7c782045b16f4af85011feebc266ca167953b82684a10bf4b Oct 31 00:16:48.438239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.438202 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:48.438340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.438291 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:16:48.624880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.624836 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" event={"ID":"f5b0197f-acd3-47bd-bb97-1f6c0760b007","Type":"ContainerStarted","Data":"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe"} Oct 31 00:16:48.624880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.624886 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" event={"ID":"f5b0197f-acd3-47bd-bb97-1f6c0760b007","Type":"ContainerStarted","Data":"d1aebb17c25dbae7c782045b16f4af85011feebc266ca167953b82684a10bf4b"} Oct 31 00:16:48.625181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.624984 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:48.626155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.626126 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerName="ta-container" probeResult="failure" output="Get \"http://10.133.0.58:8080/readyz\": dial tcp 10.133.0.58:8080: connect: connection refused" Oct 31 00:16:48.644016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:48.643963 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" podStartSLOduration=1.6439084290000001 podStartE2EDuration="1.643908429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:48.642244634 +0000 UTC m=+1047.633904221" watchObservedRunningTime="2026-10-31 00:16:48.643908429 +0000 UTC m=+1047.635568015" Oct 31 00:16:49.630996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:49.630968 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:16:51.451456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:51.451429 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:51.451953 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:51.451627 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="a8b51430-a30a-4b30-a69f-867792d4ef14" containerName="otc-container" containerID="cri-o://a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5" gracePeriod=30 Oct 31 00:16:53.810820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.810799 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:53.904389 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904311 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.904538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904390 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.904538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904425 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.904538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904444 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.904538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904466 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\" (UniqueName: \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.904538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904505 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\") pod \"a8b51430-a30a-4b30-a69f-867792d4ef14\" (UID: \"a8b51430-a30a-4b30-a69f-867792d4ef14\") " Oct 31 00:16:53.905022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.904968 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:16:53.906948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.906915 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:53.907196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.907173 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "kube-api-access-js27f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:16:53.907272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.907250 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:53.907578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.907561 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:53.907623 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:53.907591 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs" pod "a8b51430-a30a-4b30-a69f-867792d4ef14" (UID: "a8b51430-a30a-4b30-a69f-867792d4ef14"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:16:54.005604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005579 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js27f\" (UniqueName: \"kubernetes.io/projected/a8b51430-a30a-4b30-a69f-867792d4ef14-kube-api-access-js27f\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.005757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005609 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a8b51430-a30a-4b30-a69f-867792d4ef14-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.005757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005623 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.005757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005636 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.005757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005651 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.005757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.005665 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8b51430-a30a-4b30-a69f-867792d4ef14-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:16:54.650246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.650212 5577 generic.go:356] "Generic (PLEG): container finished" podID="a8b51430-a30a-4b30-a69f-867792d4ef14" containerID="a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5" exitCode=0 Oct 31 00:16:54.650418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.650280 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:16:54.650418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.650298 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a8b51430-a30a-4b30-a69f-867792d4ef14","Type":"ContainerDied","Data":"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5"} Oct 31 00:16:54.650418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.650359 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"a8b51430-a30a-4b30-a69f-867792d4ef14","Type":"ContainerDied","Data":"5ef2230e6777c9653dffbbd80d5bab19739ef138435f08bb2ce9958d002d7a57"} Oct 31 00:16:54.650418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.650389 5577 scope.go:122] "RemoveContainer" containerID="a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5" Oct 31 00:16:54.659653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.659636 5577 scope.go:122] "RemoveContainer" containerID="a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5" Oct 31 00:16:54.659948 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:16:54.659930 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5\": container with ID starting with a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5 not found: ID does not exist" containerID="a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5" Oct 31 00:16:54.659992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.659956 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5"} err="failed to get container status \"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5\": rpc error: code = NotFound desc = could not find container \"a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5\": container with ID starting with a51576e8df6f6c667da8a8ee6478d95bc952de9bbcfcd0724ec60d243b2335a5 not found: ID does not exist" Oct 31 00:16:54.671572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.671546 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:54.675426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:54.675406 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:16:55.129076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:16:55.129043 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b51430-a30a-4b30-a69f-867792d4ef14" path="/var/lib/kubelet/pods/a8b51430-a30a-4b30-a69f-867792d4ef14/volumes" Oct 31 00:16:59.292228 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:16:59.292194 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:17:25.477367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:17:25.477321 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:17:29.292531 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:17:29.292452 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:17:59.291417 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:17:59.291381 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:18:14.601231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:18:14.601195 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:25.518047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:18:25.517998 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:29.294579 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:18:29.294534 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:18:48.730997 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:18:48.730961 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:18:59.292085 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:18:59.292000 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:19:11.876161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:11.876123 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.082083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:12.082041 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.737183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:12.737131 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:12.883048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:12.883007 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:13.895783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:13.895748 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:14.174759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.174657 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:19:14.174946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.174920 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerName="ta-container" containerID="cri-o://f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe" gracePeriod=30 Oct 31 00:19:14.418846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.418818 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:19:14.477512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.477432 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\" (UniqueName: \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\") pod \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " Oct 31 00:19:14.477672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.477515 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\" (UniqueName: \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\") pod \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " Oct 31 00:19:14.477672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.477572 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\") pod \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\" (UID: \"f5b0197f-acd3-47bd-bb97-1f6c0760b007\") " Oct 31 00:19:14.477912 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.477884 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal" pod "f5b0197f-acd3-47bd-bb97-1f6c0760b007" (UID: "f5b0197f-acd3-47bd-bb97-1f6c0760b007"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:14.479687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.479663 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf" pod "f5b0197f-acd3-47bd-bb97-1f6c0760b007" (UID: "f5b0197f-acd3-47bd-bb97-1f6c0760b007"). InnerVolumeSpecName "kube-api-access-xh7cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:14.479798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.479687 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert" pod "f5b0197f-acd3-47bd-bb97-1f6c0760b007" (UID: "f5b0197f-acd3-47bd-bb97-1f6c0760b007"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:14.578810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.578779 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh7cf\" (UniqueName: \"kubernetes.io/projected/f5b0197f-acd3-47bd-bb97-1f6c0760b007-kube-api-access-xh7cf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.578810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.578805 5577 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/f5b0197f-acd3-47bd-bb97-1f6c0760b007-ta-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:14.579034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:14.578820 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/f5b0197f-acd3-47bd-bb97-1f6c0760b007-data-science-collector-ta-server-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:15.135369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135335 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135661 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a8b51430-a30a-4b30-a69f-867792d4ef14" containerName="otc-container" Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135676 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b51430-a30a-4b30-a69f-867792d4ef14" containerName="otc-container" Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135733 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerName="ta-container" Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135742 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerName="ta-container" Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135819 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerName="ta-container" Oct 31 00:19:15.135828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.135831 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b51430-a30a-4b30-a69f-867792d4ef14" containerName="otc-container" Oct 31 00:19:15.139014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.138995 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.143891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.143870 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:19:15.144033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.144019 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:19:15.144177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.144155 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:19:15.144302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.144226 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:19:15.144302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.144232 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:19:15.144302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.144296 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:19:15.149968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.149946 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:15.184851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.184826 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.184971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.184868 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.184971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.184956 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.185042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.185000 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.185088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.185069 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.185164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.185101 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnqcr\" (UniqueName: \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.254827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.254797 5577 generic.go:356] "Generic (PLEG): container finished" podID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" containerID="f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe" exitCode=0 Oct 31 00:19:15.254998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.254845 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" Oct 31 00:19:15.254998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.254849 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" event={"ID":"f5b0197f-acd3-47bd-bb97-1f6c0760b007","Type":"ContainerDied","Data":"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe"} Oct 31 00:19:15.254998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.254976 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9" event={"ID":"f5b0197f-acd3-47bd-bb97-1f6c0760b007","Type":"ContainerDied","Data":"d1aebb17c25dbae7c782045b16f4af85011feebc266ca167953b82684a10bf4b"} Oct 31 00:19:15.255165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.255006 5577 scope.go:122] "RemoveContainer" containerID="f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe" Oct 31 00:19:15.263268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.263252 5577 scope.go:122] "RemoveContainer" containerID="f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe" Oct 31 00:19:15.263484 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:19:15.263469 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe\": container with ID starting with f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe not found: ID does not exist" containerID="f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe" Oct 31 00:19:15.263537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.263494 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe"} err="failed to get container status \"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe\": rpc error: code = NotFound desc = could not find container \"f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe\": container with ID starting with f153b6a9a2e5ee0da58b75057ec52fa6cc924fee71064cee019a938dcda1affe not found: ID does not exist" Oct 31 00:19:15.274230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.274206 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:19:15.280563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.280545 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-db564bcf8-vs7q9"] Oct 31 00:19:15.286108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286092 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.286173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286136 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.286173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286169 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.286282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286189 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.286282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286218 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.286282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286244 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnqcr\" (UniqueName: \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.287014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.286962 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.288899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.288879 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.288979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.288950 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.289253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.289234 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.289308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.289295 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.299483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.299461 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnqcr\" (UniqueName: \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\") pod \"data-science-collector-collector-1\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.451072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.450981 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:15.732780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.732695 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:15.736439 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:19:15.736416 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3ad1011_c7df_41ea_9577_023ad207cb01.slice/crio-39fc545ac284acb2923f1c058895408f14bfb116cd7cb09e7e4cae72c3c48426 WatchSource:0}: Error finding container 39fc545ac284acb2923f1c058895408f14bfb116cd7cb09e7e4cae72c3c48426: Status 404 returned error can't find the container with id 39fc545ac284acb2923f1c058895408f14bfb116cd7cb09e7e4cae72c3c48426 Oct 31 00:19:15.738550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.738521 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:15.738633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:15.738579 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:16.260611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:16.260579 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3ad1011-c7df-41ea-9577-023ad207cb01","Type":"ContainerStarted","Data":"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6"} Oct 31 00:19:16.261085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:16.260629 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3ad1011-c7df-41ea-9577-023ad207cb01","Type":"ContainerStarted","Data":"39fc545ac284acb2923f1c058895408f14bfb116cd7cb09e7e4cae72c3c48426"} Oct 31 00:19:16.282362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:16.282303 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.282285421 podStartE2EDuration="1.282285421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:16.28132403 +0000 UTC m=+1195.272983632" watchObservedRunningTime="2026-10-31 00:19:16.282285421 +0000 UTC m=+1195.273945006" Oct 31 00:19:17.129663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:17.129628 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b0197f-acd3-47bd-bb97-1f6c0760b007" path="/var/lib/kubelet/pods/f5b0197f-acd3-47bd-bb97-1f6c0760b007/volumes" Oct 31 00:19:18.994162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:18.994088 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:21.250750 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:21.250720 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:19:21.252593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:21.252571 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:19:24.592259 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:24.592219 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:26.039283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:26.039248 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:29.293846 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:19:29.293805 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:19:35.261001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:35.260964 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.399417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:35.399374 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:35.805745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:35.805698 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:35.805926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:35.805908 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d3ad1011-c7df-41ea-9577-023ad207cb01" containerName="otc-container" containerID="cri-o://424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6" gracePeriod=30 Oct 31 00:19:35.928106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:35.928043 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:36.089844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:36.089744 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:36.466702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:36.466511 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:36.681691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:36.681653 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:36.879820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:36.879544 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:19:37.061363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.061342 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.175313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175229 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.175313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175290 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.175512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175397 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.175512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175440 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\" (UniqueName: \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.175512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175473 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.175675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.175543 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\") pod \"d3ad1011-c7df-41ea-9577-023ad207cb01\" (UID: \"d3ad1011-c7df-41ea-9577-023ad207cb01\") " Oct 31 00:19:37.176069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.176039 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:37.177880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.177841 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.177989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.177922 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.178399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.178371 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.178478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.178436 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:19:37.178561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.178543 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr" pod "d3ad1011-c7df-41ea-9577-023ad207cb01" (UID: "d3ad1011-c7df-41ea-9577-023ad207cb01"). InnerVolumeSpecName "kube-api-access-pnqcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:37.277213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277177 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.277369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277219 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.277369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277236 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.277369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277252 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnqcr\" (UniqueName: \"kubernetes.io/projected/d3ad1011-c7df-41ea-9577-023ad207cb01-kube-api-access-pnqcr\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.277369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277267 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3ad1011-c7df-41ea-9577-023ad207cb01-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.277369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.277280 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3ad1011-c7df-41ea-9577-023ad207cb01-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:37.355140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.355102 5577 generic.go:356] "Generic (PLEG): container finished" podID="d3ad1011-c7df-41ea-9577-023ad207cb01" containerID="424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6" exitCode=0 Oct 31 00:19:37.355286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.355168 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.355286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.355183 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3ad1011-c7df-41ea-9577-023ad207cb01","Type":"ContainerDied","Data":"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6"} Oct 31 00:19:37.355286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.355263 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3ad1011-c7df-41ea-9577-023ad207cb01","Type":"ContainerDied","Data":"39fc545ac284acb2923f1c058895408f14bfb116cd7cb09e7e4cae72c3c48426"} Oct 31 00:19:37.355286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.355285 5577 scope.go:122] "RemoveContainer" containerID="424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6" Oct 31 00:19:37.364312 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.364295 5577 scope.go:122] "RemoveContainer" containerID="424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6" Oct 31 00:19:37.364586 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:19:37.364568 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6\": container with ID starting with 424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6 not found: ID does not exist" containerID="424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6" Oct 31 00:19:37.364627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.364594 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6"} err="failed to get container status \"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6\": rpc error: code = NotFound desc = could not find container \"424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6\": container with ID starting with 424ae3b00e220cee5526496d678d4a5c7e60924de5eb803d5242b7b65bf86bf6 not found: ID does not exist" Oct 31 00:19:37.381758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.381699 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:37.386942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.386921 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:37.414845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.414811 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:37.415156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.415145 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d3ad1011-c7df-41ea-9577-023ad207cb01" containerName="otc-container" Oct 31 00:19:37.415207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.415158 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3ad1011-c7df-41ea-9577-023ad207cb01" containerName="otc-container" Oct 31 00:19:37.415245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.415241 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3ad1011-c7df-41ea-9577-023ad207cb01" containerName="otc-container" Oct 31 00:19:37.419585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.419566 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.423857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.423837 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:19:37.423993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.423864 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:19:37.423993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.423884 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:19:37.423993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.423953 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:19:37.424268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.424250 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:19:37.424268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.424262 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:19:37.428624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.428603 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:37.479321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479287 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.479321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479326 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpkzd\" (UniqueName: \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.479861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479348 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.479861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479466 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.479861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479517 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.479861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.479598 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.580923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.580885 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.581113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.580935 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.581113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.580960 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.581113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.581069 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpkzd\" (UniqueName: \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.581113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.581110 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.581322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.581138 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.582045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.582016 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.583850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.583820 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.584058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.584042 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.584237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.584217 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.584344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.584217 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.595219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.595198 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpkzd\" (UniqueName: \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\") pod \"data-science-collector-collector-1\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:37.732870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:37.732782 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:19:38.007620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.007590 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:19:38.010938 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:19:38.010912 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8facbc32_e1aa_4a0e_b6ae_f0b5ec106e55.slice/crio-1d5153be6003e1a6cf41058f118d9c5ac6167472c176722304194403dfd2b271 WatchSource:0}: Error finding container 1d5153be6003e1a6cf41058f118d9c5ac6167472c176722304194403dfd2b271: Status 404 returned error can't find the container with id 1d5153be6003e1a6cf41058f118d9c5ac6167472c176722304194403dfd2b271 Oct 31 00:19:38.013110 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.013081 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:38.013212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.013153 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:19:38.361144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.361045 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55","Type":"ContainerStarted","Data":"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a"} Oct 31 00:19:38.361144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.361102 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55","Type":"ContainerStarted","Data":"1d5153be6003e1a6cf41058f118d9c5ac6167472c176722304194403dfd2b271"} Oct 31 00:19:38.382040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:38.381990 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.3819764079999999 podStartE2EDuration="1.381976408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:38.381472274 +0000 UTC m=+1217.373131883" watchObservedRunningTime="2026-10-31 00:19:38.381976408 +0000 UTC m=+1217.373636024" Oct 31 00:19:39.130264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:19:39.130223 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3ad1011-c7df-41ea-9577-023ad207cb01" path="/var/lib/kubelet/pods/d3ad1011-c7df-41ea-9577-023ad207cb01/volumes" Oct 31 00:19:59.294013 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:19:59.293975 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:20:25.631520 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:20:25.631482 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:20:29.292451 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:20:29.292372 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:20:59.291177 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:20:59.291141 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:21:21.365884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:21:21.365851 5577 scope.go:122] "RemoveContainer" containerID="66426cc63ecc5d62f9a2b8282c510936ed1a04b2fe9b1b8a1f04495a0ff94f22" Oct 31 00:21:25.687279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:21:25.687237 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:29.294568 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:21:29.294528 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:21:42.927384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:21:42.927344 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:21:59.291851 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:21:59.291776 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:22:25.744025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:22:25.743979 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:29.294918 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:22:29.294886 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:22:38.991787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:22:38.991750 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:39.119437 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:22:39.119398 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:22:59.291697 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:22:59.291659 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:23:25.147327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:25.147292 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:25.796917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:25.796877 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:29.293194 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:23:29.293121 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:23:31.346472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:31.346432 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:31.486907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:31.486870 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:33.351589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:33.351552 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:33.488276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:33.488234 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:41.109835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:41.109795 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:41.252427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:41.252390 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:45.402640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:45.402606 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:45.537400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:45.537357 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:52.861882 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:52.861842 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:53.106732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:53.106668 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:53.419428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:53.419396 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:53.419602 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:53.419585 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" containerName="otc-container" containerID="cri-o://6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a" gracePeriod=30 Oct 31 00:23:53.539665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:53.539630 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:53.666566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:53.666531 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:23:55.757931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.757906 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:55.839324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839249 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\" (UniqueName: \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839296 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839331 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839353 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839382 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839417 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\") pod \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\" (UID: \"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55\") " Oct 31 00:23:55.839797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.839775 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:23:55.841742 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.841690 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "kube-api-access-fpkzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:23:55.841742 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.841690 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:55.841935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.841775 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:55.842097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.842072 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:55.842148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.842117 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs" pod "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" (UID: "8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:23:55.940456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940422 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.940456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940448 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpkzd\" (UniqueName: \"kubernetes.io/projected/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-kube-api-access-fpkzd\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.940456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940457 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.940456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940466 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.940733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940476 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:55.940733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:55.940484 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:23:56.453438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.453402 5577 generic.go:356] "Generic (PLEG): container finished" podID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" containerID="6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a" exitCode=0 Oct 31 00:23:56.453621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.453452 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55","Type":"ContainerDied","Data":"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a"} Oct 31 00:23:56.453621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.453456 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.453621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.453492 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55","Type":"ContainerDied","Data":"1d5153be6003e1a6cf41058f118d9c5ac6167472c176722304194403dfd2b271"} Oct 31 00:23:56.453621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.453514 5577 scope.go:122] "RemoveContainer" containerID="6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a" Oct 31 00:23:56.462436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.462419 5577 scope.go:122] "RemoveContainer" containerID="6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a" Oct 31 00:23:56.462680 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:23:56.462659 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a\": container with ID starting with 6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a not found: ID does not exist" containerID="6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a" Oct 31 00:23:56.462759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.462694 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a"} err="failed to get container status \"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a\": rpc error: code = NotFound desc = could not find container \"6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a\": container with ID starting with 6dc7f5c84e6891446bcde8276e84a389eb9dd08f2cb8427d6d73237d6daaf16a not found: ID does not exist" Oct 31 00:23:56.475051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.475028 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:56.479405 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.479383 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:56.505935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.505912 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:56.506237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.506225 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" containerName="otc-container" Oct 31 00:23:56.506280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.506239 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" containerName="otc-container" Oct 31 00:23:56.506314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.506309 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" containerName="otc-container" Oct 31 00:23:56.510476 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.510461 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.514486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514470 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:23:56.514839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514821 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:23:56.514928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514857 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-g8x7h\"" Oct 31 00:23:56.514928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514868 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:23:56.514928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514877 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:23:56.515056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.514950 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:23:56.519991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.519970 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:56.546253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546224 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.546351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546275 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.546402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546346 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.546402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546395 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxg6z\" (UniqueName: \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.546512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546415 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.546566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.546517 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647395 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647443 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647475 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647523 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647553 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.647620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.647607 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxg6z\" (UniqueName: \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.648334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.648235 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.650205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.650183 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.650310 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.650241 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.650502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.650485 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.650613 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.650593 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.661663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.661643 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxg6z\" (UniqueName: \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\") pod \"data-science-collector-collector-1\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:56.824126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:56.824086 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:23:57.104854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.104770 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:23:57.110950 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:23:57.110865 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec664d8b_96f3_4907_83cc_647a27ff7fda.slice/crio-b5ce1035029c16d74f25b9856320692b33bd1d7c77ea72e28afb3b867a80462c WatchSource:0}: Error finding container b5ce1035029c16d74f25b9856320692b33bd1d7c77ea72e28afb3b867a80462c: Status 404 returned error can't find the container with id b5ce1035029c16d74f25b9856320692b33bd1d7c77ea72e28afb3b867a80462c Oct 31 00:23:57.112776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.112748 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:23:57.112862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.112808 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:23:57.131183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.131157 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55" path="/var/lib/kubelet/pods/8facbc32-e1aa-4a0e-b6ae-f0b5ec106e55/volumes" Oct 31 00:23:57.460263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.460173 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec664d8b-96f3-4907-83cc-647a27ff7fda","Type":"ContainerStarted","Data":"aa11fed6e590d634a1482f62810d53f685a58c782d0e3f0953e8550aca277d0b"} Oct 31 00:23:57.460432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.460276 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec664d8b-96f3-4907-83cc-647a27ff7fda","Type":"ContainerStarted","Data":"b5ce1035029c16d74f25b9856320692b33bd1d7c77ea72e28afb3b867a80462c"} Oct 31 00:23:57.481948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:23:57.481898 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.4818846620000001 podStartE2EDuration="1.481884662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:57.480657447 +0000 UTC m=+1476.472317031" watchObservedRunningTime="2026-10-31 00:23:57.481884662 +0000 UTC m=+1476.473544246" Oct 31 00:23:59.295297 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:23:59.295255 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:24:15.365782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.365740 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0"] Oct 31 00:24:15.369600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.369578 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" Oct 31 00:24:15.375751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.375723 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0"] Oct 31 00:24:15.383770 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.383744 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" Oct 31 00:24:15.662731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.662607 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0"] Oct 31 00:24:15.666128 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:24:15.666091 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbd9ecae0_1bdb_4f17_aa84_f8753dbb089b.slice/crio-109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402 WatchSource:0}: Error finding container 109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402: Status 404 returned error can't find the container with id 109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402 Oct 31 00:24:15.668046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:15.668022 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:24:16.546049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:16.546017 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" event={"ID":"bd9ecae0-1bdb-4f17-aa84-f8753dbb089b","Type":"ContainerStarted","Data":"109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402"} Oct 31 00:24:21.279795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:21.279768 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:24:21.280559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:21.280539 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:24:23.580292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:23.580260 5577 generic.go:356] "Generic (PLEG): container finished" podID="bd9ecae0-1bdb-4f17-aa84-f8753dbb089b" containerID="14dd9b704f3751e92bbc1155d0dac04d5348d85518f293086646b72302fc2a15" exitCode=0 Oct 31 00:24:23.580646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:23.580349 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" event={"ID":"bd9ecae0-1bdb-4f17-aa84-f8753dbb089b","Type":"ContainerDied","Data":"14dd9b704f3751e92bbc1155d0dac04d5348d85518f293086646b72302fc2a15"} Oct 31 00:24:24.710547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:24.710524 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" Oct 31 00:24:25.591541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:25.591502 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" Oct 31 00:24:25.591541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:25.591521 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0" event={"ID":"bd9ecae0-1bdb-4f17-aa84-f8753dbb089b","Type":"ContainerDied","Data":"109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402"} Oct 31 00:24:25.591826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:25.591551 5577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="109702c00c564cc23dffbcfb180357385dd44ff5750d0a87dda5f5bb84947402" Oct 31 00:24:25.853820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:25.853725 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:29.293080 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:24:29.293046 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:24:39.866679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:39.866640 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:41.154771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:41.154733 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:51.948863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:24:51.948826 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:24:59.292577 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:24:59.292494 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:25:25.906083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:25:25.906047 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:29.292306 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:25:29.292269 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:25:38.874644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:25:38.874608 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:25:59.291996 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:25:59.291959 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:26:26.000252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:26:26.000219 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:26:29.294471 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:26:29.294386 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:26:59.292623 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:26:59.292584 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:27:26.024441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:27:26.024398 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:27:29.293400 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:27:29.293363 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:27:59.292940 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:27:59.292856 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:28:09.335525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:09.335490 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0"] Oct 31 00:28:09.341519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:09.341492 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/collector-prometheus-allowed-18e3757cc0ee38d0"] Oct 31 00:28:09.418618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:09.418583 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:09.552087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:09.552046 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:11.129384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:11.129351 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9ecae0-1bdb-4f17-aa84-f8753dbb089b" path="/var/lib/kubelet/pods/bd9ecae0-1bdb-4f17-aa84-f8753dbb089b/volumes" Oct 31 00:28:16.034444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:16.034404 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:16.163756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:16.163700 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.627700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:20.627650 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:20.794844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:20.794795 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:25.120946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:25.120913 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:26.075244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:26.075200 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:29.292904 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:29.292872 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:28:35.473747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:35.473693 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.016435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.016396 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:28:41.016920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.016679 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="kube-rbac-proxy" containerID="cri-o://c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" gracePeriod=30 Oct 31 00:28:41.016920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.016750 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="prom-label-proxy" containerID="cri-o://9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" gracePeriod=30 Oct 31 00:28:41.016920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.016789 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="method-gate" containerID="cri-o://f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" gracePeriod=30 Oct 31 00:28:41.021207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.021174 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:28:41.021642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.021416 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="kube-rbac-proxy" containerID="cri-o://8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" gracePeriod=30 Oct 31 00:28:41.021642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.021450 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="prom-label-proxy" containerID="cri-o://d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" gracePeriod=30 Oct 31 00:28:41.021642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.021462 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="method-gate" containerID="cri-o://c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" gracePeriod=30 Oct 31 00:28:41.247433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.247394 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.261538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.261513 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:28:41.261788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.261757 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" containerID="cri-o://96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2" gracePeriod=120 Oct 31 00:28:41.261848 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.261809 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="config-reloader" containerID="cri-o://377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8" gracePeriod=120 Oct 31 00:28:41.335055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.335028 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:28:41.337343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.337325 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:28:41.427318 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.427246 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.482691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.482661 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:41.482930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.482907 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ec664d8b-96f3-4907-83cc-647a27ff7fda" containerName="otc-container" containerID="cri-o://aa11fed6e590d634a1482f62810d53f685a58c782d0e3f0953e8550aca277d0b" gracePeriod=30 Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505344 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\") pod \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505409 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\" (UniqueName: \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\") pod \"76090f72-3199-4646-ab80-8809a4284b0f\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505445 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\") pod \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505478 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\" (UniqueName: \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\") pod \"76090f72-3199-4646-ab80-8809a4284b0f\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505526 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") pod \"76090f72-3199-4646-ab80-8809a4284b0f\" (UID: \"76090f72-3199-4646-ab80-8809a4284b0f\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505557 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\") pod \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505661 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\" (UniqueName: \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\") pod \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505694 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\" (UniqueName: \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\") pod \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\" (UID: \"da85f980-ec6f-4c5a-afc2-b84ed71ac9da\") " Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505753 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config" pod "da85f980-ec6f-4c5a-afc2-b84ed71ac9da" (UID: "da85f980-ec6f-4c5a-afc2-b84ed71ac9da"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505925 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca" pod "da85f980-ec6f-4c5a-afc2-b84ed71ac9da" (UID: "da85f980-ec6f-4c5a-afc2-b84ed71ac9da"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.505980 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.506259 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config" pod "76090f72-3199-4646-ab80-8809a4284b0f" (UID: "76090f72-3199-4646-ab80-8809a4284b0f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.506303 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca" pod "da85f980-ec6f-4c5a-afc2-b84ed71ac9da" (UID: "da85f980-ec6f-4c5a-afc2-b84ed71ac9da"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.509177 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2" pod "da85f980-ec6f-4c5a-afc2-b84ed71ac9da" (UID: "da85f980-ec6f-4c5a-afc2-b84ed71ac9da"). InnerVolumeSpecName "kube-api-access-5m9p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.509310 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96" pod "76090f72-3199-4646-ab80-8809a4284b0f" (UID: "76090f72-3199-4646-ab80-8809a4284b0f"). InnerVolumeSpecName "kube-api-access-knb96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.509316 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls" pod "da85f980-ec6f-4c5a-afc2-b84ed71ac9da" (UID: "da85f980-ec6f-4c5a-afc2-b84ed71ac9da"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.516744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.509319 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls" pod "76090f72-3199-4646-ab80-8809a4284b0f" (UID: "76090f72-3199-4646-ab80-8809a4284b0f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.607102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.606932 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76090f72-3199-4646-ab80-8809a4284b0f-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607112 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607130 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knb96\" (UniqueName: \"kubernetes.io/projected/76090f72-3199-4646-ab80-8809a4284b0f-kube-api-access-knb96\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607144 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/76090f72-3199-4646-ab80-8809a4284b0f-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607161 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607176 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m9p2\" (UniqueName: \"kubernetes.io/projected/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-kube-api-access-5m9p2\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.607243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.607185 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/da85f980-ec6f-4c5a-afc2-b84ed71ac9da-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:41.681177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.681144 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:41.681380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.681346 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="dfadc60c-8b53-4331-91d7-7b131492017f" containerName="perses" containerID="cri-o://80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d" gracePeriod=30 Oct 31 00:28:41.682536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.682510 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.692497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.692468 5577 generic.go:356] "Generic (PLEG): container finished" podID="ec664d8b-96f3-4907-83cc-647a27ff7fda" containerID="aa11fed6e590d634a1482f62810d53f685a58c782d0e3f0953e8550aca277d0b" exitCode=0 Oct 31 00:28:41.692744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.692565 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec664d8b-96f3-4907-83cc-647a27ff7fda","Type":"ContainerDied","Data":"aa11fed6e590d634a1482f62810d53f685a58c782d0e3f0953e8550aca277d0b"} Oct 31 00:28:41.696284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.695303 5577 generic.go:356] "Generic (PLEG): container finished" podID="76090f72-3199-4646-ab80-8809a4284b0f" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" exitCode=0 Oct 31 00:28:41.696284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696270 5577 generic.go:356] "Generic (PLEG): container finished" podID="76090f72-3199-4646-ab80-8809a4284b0f" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" exitCode=0 Oct 31 00:28:41.696284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696285 5577 generic.go:356] "Generic (PLEG): container finished" podID="76090f72-3199-4646-ab80-8809a4284b0f" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" exitCode=0 Oct 31 00:28:41.696505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696405 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerDied","Data":"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6"} Oct 31 00:28:41.696505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696417 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" Oct 31 00:28:41.696505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696474 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerDied","Data":"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4"} Oct 31 00:28:41.696637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696514 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerDied","Data":"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4"} Oct 31 00:28:41.696637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696593 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn" event={"ID":"76090f72-3199-4646-ab80-8809a4284b0f","Type":"ContainerDied","Data":"c228d1496a5db06b8ee0f776762605fdbaa02b2eb96b24ce8d246c4cd636b38e"} Oct 31 00:28:41.696775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.696637 5577 scope.go:122] "RemoveContainer" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" Oct 31 00:28:41.710671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.710636 5577 generic.go:356] "Generic (PLEG): container finished" podID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerID="377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8" exitCode=0 Oct 31 00:28:41.710874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.710776 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerDied","Data":"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8"} Oct 31 00:28:41.713790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713763 5577 generic.go:356] "Generic (PLEG): container finished" podID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" exitCode=0 Oct 31 00:28:41.713790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713788 5577 generic.go:356] "Generic (PLEG): container finished" podID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" exitCode=0 Oct 31 00:28:41.713956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713797 5577 generic.go:356] "Generic (PLEG): container finished" podID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" exitCode=0 Oct 31 00:28:41.713956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713850 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerDied","Data":"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6"} Oct 31 00:28:41.713956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713940 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" Oct 31 00:28:41.714195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.713942 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerDied","Data":"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4"} Oct 31 00:28:41.714269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.714223 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerDied","Data":"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc"} Oct 31 00:28:41.714345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.714267 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj" event={"ID":"da85f980-ec6f-4c5a-afc2-b84ed71ac9da","Type":"ContainerDied","Data":"26aa62af96814baba023d6f1064caf9a44b1715af58db2d7b1652ac0b4c00273"} Oct 31 00:28:41.763060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.763037 5577 scope.go:122] "RemoveContainer" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" Oct 31 00:28:41.778447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.778421 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:28:41.785677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.785654 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-z5htn"] Oct 31 00:28:41.798045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.798021 5577 scope.go:122] "RemoveContainer" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" Oct 31 00:28:41.806639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.806614 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:41.807541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.807464 5577 scope.go:122] "RemoveContainer" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" Oct 31 00:28:41.809031 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.808891 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": container with ID starting with d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6 not found: ID does not exist" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" Oct 31 00:28:41.809031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.808930 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6"} err="failed to get container status \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": rpc error: code = NotFound desc = could not find container \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": container with ID starting with d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6 not found: ID does not exist" Oct 31 00:28:41.809031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.808952 5577 scope.go:122] "RemoveContainer" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" Oct 31 00:28:41.809241 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.809173 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": container with ID starting with c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4 not found: ID does not exist" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" Oct 31 00:28:41.809241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809199 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4"} err="failed to get container status \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": rpc error: code = NotFound desc = could not find container \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": container with ID starting with c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4 not found: ID does not exist" Oct 31 00:28:41.809241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809215 5577 scope.go:122] "RemoveContainer" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" Oct 31 00:28:41.809241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809215 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:28:41.809638 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.809621 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": container with ID starting with 8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4 not found: ID does not exist" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" Oct 31 00:28:41.809760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809640 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4"} err="failed to get container status \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": rpc error: code = NotFound desc = could not find container \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": container with ID starting with 8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4 not found: ID does not exist" Oct 31 00:28:41.809760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809656 5577 scope.go:122] "RemoveContainer" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" Oct 31 00:28:41.809957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809938 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6"} err="failed to get container status \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": rpc error: code = NotFound desc = could not find container \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": container with ID starting with d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6 not found: ID does not exist" Oct 31 00:28:41.810037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.809969 5577 scope.go:122] "RemoveContainer" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" Oct 31 00:28:41.810338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810317 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4"} err="failed to get container status \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": rpc error: code = NotFound desc = could not find container \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": container with ID starting with c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4 not found: ID does not exist" Oct 31 00:28:41.810426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810340 5577 scope.go:122] "RemoveContainer" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" Oct 31 00:28:41.810639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810604 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4"} err="failed to get container status \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": rpc error: code = NotFound desc = could not find container \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": container with ID starting with 8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4 not found: ID does not exist" Oct 31 00:28:41.810639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810628 5577 scope.go:122] "RemoveContainer" containerID="d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6" Oct 31 00:28:41.810905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810887 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6"} err="failed to get container status \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": rpc error: code = NotFound desc = could not find container \"d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6\": container with ID starting with d6cb95d9b68737b555217c62deb3a31a71f005aa36d1d340f10a1c8c348e11f6 not found: ID does not exist" Oct 31 00:28:41.810963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.810906 5577 scope.go:122] "RemoveContainer" containerID="c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4" Oct 31 00:28:41.811189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.811171 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4"} err="failed to get container status \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": rpc error: code = NotFound desc = could not find container \"c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4\": container with ID starting with c0af767ef7ecaabac11879d73032676b4b16206025adc0b24da3c8455bcd46b4 not found: ID does not exist" Oct 31 00:28:41.811189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.811187 5577 scope.go:122] "RemoveContainer" containerID="8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4" Oct 31 00:28:41.811447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.811428 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4"} err="failed to get container status \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": rpc error: code = NotFound desc = could not find container \"8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4\": container with ID starting with 8742e956cfd38eeba0290791beb1c2072d780ecde4b5d5be8b6e97301457d6b4 not found: ID does not exist" Oct 31 00:28:41.811528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.811456 5577 scope.go:122] "RemoveContainer" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" Oct 31 00:28:41.813491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.813471 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-wbwrj"] Oct 31 00:28:41.822683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.822661 5577 scope.go:122] "RemoveContainer" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" Oct 31 00:28:41.831495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.831468 5577 scope.go:122] "RemoveContainer" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" Oct 31 00:28:41.845353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.845322 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:41.846243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.846223 5577 scope.go:122] "RemoveContainer" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" Oct 31 00:28:41.846520 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.846500 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": container with ID starting with 9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6 not found: ID does not exist" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" Oct 31 00:28:41.846589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.846532 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6"} err="failed to get container status \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": rpc error: code = NotFound desc = could not find container \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": container with ID starting with 9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6 not found: ID does not exist" Oct 31 00:28:41.846589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.846557 5577 scope.go:122] "RemoveContainer" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" Oct 31 00:28:41.846837 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.846817 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": container with ID starting with f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4 not found: ID does not exist" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" Oct 31 00:28:41.846938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.846842 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4"} err="failed to get container status \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": rpc error: code = NotFound desc = could not find container \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": container with ID starting with f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4 not found: ID does not exist" Oct 31 00:28:41.846938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.846858 5577 scope.go:122] "RemoveContainer" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" Oct 31 00:28:41.847069 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:41.847035 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": container with ID starting with c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc not found: ID does not exist" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" Oct 31 00:28:41.847069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847051 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc"} err="failed to get container status \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": rpc error: code = NotFound desc = could not find container \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": container with ID starting with c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc not found: ID does not exist" Oct 31 00:28:41.847069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847064 5577 scope.go:122] "RemoveContainer" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" Oct 31 00:28:41.847234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847214 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6"} err="failed to get container status \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": rpc error: code = NotFound desc = could not find container \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": container with ID starting with 9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6 not found: ID does not exist" Oct 31 00:28:41.847234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847226 5577 scope.go:122] "RemoveContainer" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" Oct 31 00:28:41.847563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847380 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4"} err="failed to get container status \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": rpc error: code = NotFound desc = could not find container \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": container with ID starting with f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4 not found: ID does not exist" Oct 31 00:28:41.847563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847396 5577 scope.go:122] "RemoveContainer" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" Oct 31 00:28:41.847563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847542 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc"} err="failed to get container status \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": rpc error: code = NotFound desc = could not find container \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": container with ID starting with c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc not found: ID does not exist" Oct 31 00:28:41.847563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847554 5577 scope.go:122] "RemoveContainer" containerID="9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6" Oct 31 00:28:41.848429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847729 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6"} err="failed to get container status \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": rpc error: code = NotFound desc = could not find container \"9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6\": container with ID starting with 9a92ef2d10d3f852b944c3aac2de8fdf07731443463941cd7797faff2f8a03d6 not found: ID does not exist" Oct 31 00:28:41.848429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847748 5577 scope.go:122] "RemoveContainer" containerID="f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4" Oct 31 00:28:41.848429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847969 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4"} err="failed to get container status \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": rpc error: code = NotFound desc = could not find container \"f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4\": container with ID starting with f89e3bb9865fb4f32d45eb7ad72d2c0db2bc270521d9d25d3b711d159034b9a4 not found: ID does not exist" Oct 31 00:28:41.848429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.847984 5577 scope.go:122] "RemoveContainer" containerID="c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc" Oct 31 00:28:41.848429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.848130 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc"} err="failed to get container status \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": rpc error: code = NotFound desc = could not find container \"c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc\": container with ID starting with c1e284f2931f830f3ed2561f62e02be22410da3d675cb39d4c29d68e72af20dc not found: ID does not exist" Oct 31 00:28:41.910107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910079 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\" (UniqueName: \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910122 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910165 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910232 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910280 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910321 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\") pod \"ec664d8b-96f3-4907-83cc-647a27ff7fda\" (UID: \"ec664d8b-96f3-4907-83cc-647a27ff7fda\") " Oct 31 00:28:41.910701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.910669 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:41.913254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.913221 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.913354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.913258 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "kube-api-access-qxg6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:41.913354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.913258 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.913354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.913274 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.913354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.913342 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs" pod "ec664d8b-96f3-4907-83cc-647a27ff7fda" (UID: "ec664d8b-96f3-4907-83cc-647a27ff7fda"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:41.936328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.936306 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:41.982299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:41.982271 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.011450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011419 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ec664d8b-96f3-4907-83cc-647a27ff7fda-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.011450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011447 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.011617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011457 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.011617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011466 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxg6z\" (UniqueName: \"kubernetes.io/projected/ec664d8b-96f3-4907-83cc-647a27ff7fda-kube-api-access-qxg6z\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.011617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011475 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.011617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.011483 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ec664d8b-96f3-4907-83cc-647a27ff7fda-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.112480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.112414 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\" (UniqueName: \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\") pod \"dfadc60c-8b53-4331-91d7-7b131492017f\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " Oct 31 00:28:42.112930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.112500 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\" (UniqueName: \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\") pod \"dfadc60c-8b53-4331-91d7-7b131492017f\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " Oct 31 00:28:42.112930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.112522 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\" (UniqueName: \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\") pod \"dfadc60c-8b53-4331-91d7-7b131492017f\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " Oct 31 00:28:42.112930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.112665 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"dfadc60c-8b53-4331-91d7-7b131492017f\" (UID: \"dfadc60c-8b53-4331-91d7-7b131492017f\") " Oct 31 00:28:42.113051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.112933 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config" pod "dfadc60c-8b53-4331-91d7-7b131492017f" (UID: "dfadc60c-8b53-4331-91d7-7b131492017f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:42.114908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.114878 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w" pod "dfadc60c-8b53-4331-91d7-7b131492017f" (UID: "dfadc60c-8b53-4331-91d7-7b131492017f"). InnerVolumeSpecName "kube-api-access-m497w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:42.115562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.115538 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "dfadc60c-8b53-4331-91d7-7b131492017f" (UID: "dfadc60c-8b53-4331-91d7-7b131492017f"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:28:42.213994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.213956 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:28:42.213994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.213990 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m497w\" (UniqueName: \"kubernetes.io/projected/dfadc60c-8b53-4331-91d7-7b131492017f-kube-api-access-m497w\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.213994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.214002 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/dfadc60c-8b53-4331-91d7-7b131492017f-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.227350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.227333 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:28:42.261500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.261469 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins" pod "dfadc60c-8b53-4331-91d7-7b131492017f" (UID: "dfadc60c-8b53-4331-91d7-7b131492017f"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:42.314698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.314663 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/dfadc60c-8b53-4331-91d7-7b131492017f-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:42.314698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.314692 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:28:42.719355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.719315 5577 generic.go:356] "Generic (PLEG): container finished" podID="dfadc60c-8b53-4331-91d7-7b131492017f" containerID="80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d" exitCode=0 Oct 31 00:28:42.719355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.719359 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:42.719618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.719403 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"dfadc60c-8b53-4331-91d7-7b131492017f","Type":"ContainerDied","Data":"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d"} Oct 31 00:28:42.719618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.719449 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"dfadc60c-8b53-4331-91d7-7b131492017f","Type":"ContainerDied","Data":"91e24faf4d3ce595199e78f6304d0685493635061d32b77799b3b259a4a05d71"} Oct 31 00:28:42.719618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.719473 5577 scope.go:122] "RemoveContainer" containerID="80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d" Oct 31 00:28:42.720933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.720916 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:28:42.721022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.720930 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ec664d8b-96f3-4907-83cc-647a27ff7fda","Type":"ContainerDied","Data":"b5ce1035029c16d74f25b9856320692b33bd1d7c77ea72e28afb3b867a80462c"} Oct 31 00:28:42.735534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.735512 5577 scope.go:122] "RemoveContainer" containerID="80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d" Oct 31 00:28:42.735863 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:42.735839 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d\": container with ID starting with 80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d not found: ID does not exist" containerID="80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d" Oct 31 00:28:42.735955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.735869 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d"} err="failed to get container status \"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d\": rpc error: code = NotFound desc = could not find container \"80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d\": container with ID starting with 80b141ad85a8ff15d2ae1d0aef8e180aca74a6ea5e47808472fe211bbf74011d not found: ID does not exist" Oct 31 00:28:42.735955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.735888 5577 scope.go:122] "RemoveContainer" containerID="aa11fed6e590d634a1482f62810d53f685a58c782d0e3f0953e8550aca277d0b" Oct 31 00:28:42.748472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.748431 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:42.752902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.752882 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:42.764595 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.764575 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:42.769272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.769250 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:28:42.825931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.825898 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:42.845410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.845373 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.43:9093/-/ready\": dial tcp 10.133.0.43:9093: connect: connection refused" Oct 31 00:28:42.965860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:42.965823 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:43.129196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:43.129162 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76090f72-3199-4646-ab80-8809a4284b0f" path="/var/lib/kubelet/pods/76090f72-3199-4646-ab80-8809a4284b0f/volumes" Oct 31 00:28:43.129658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:43.129643 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" path="/var/lib/kubelet/pods/da85f980-ec6f-4c5a-afc2-b84ed71ac9da/volumes" Oct 31 00:28:43.130473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:43.130458 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfadc60c-8b53-4331-91d7-7b131492017f" path="/var/lib/kubelet/pods/dfadc60c-8b53-4331-91d7-7b131492017f/volumes" Oct 31 00:28:43.130864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:43.130851 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec664d8b-96f3-4907-83cc-647a27ff7fda" path="/var/lib/kubelet/pods/ec664d8b-96f3-4907-83cc-647a27ff7fda/volumes" Oct 31 00:28:44.573871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.573829 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.706132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.706103 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.830401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830320 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:44.830697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830679 5577 cpu_manager.go:414] "containerMap: removing container" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="method-gate" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830701 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="method-gate" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830763 5577 cpu_manager.go:414] "containerMap: removing container" podUID="dfadc60c-8b53-4331-91d7-7b131492017f" containerName="perses" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830773 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfadc60c-8b53-4331-91d7-7b131492017f" containerName="perses" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830800 5577 cpu_manager.go:414] "containerMap: removing container" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="kube-rbac-proxy" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830807 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="kube-rbac-proxy" Oct 31 00:28:44.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830819 5577 cpu_manager.go:414] "containerMap: removing container" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="method-gate" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830826 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="method-gate" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830836 5577 cpu_manager.go:414] "containerMap: removing container" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830843 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830861 5577 cpu_manager.go:414] "containerMap: removing container" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830868 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830882 5577 cpu_manager.go:414] "containerMap: removing container" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="kube-rbac-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830890 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="kube-rbac-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830906 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ec664d8b-96f3-4907-83cc-647a27ff7fda" containerName="otc-container" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830912 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec664d8b-96f3-4907-83cc-647a27ff7fda" containerName="otc-container" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830928 5577 cpu_manager.go:414] "containerMap: removing container" podUID="bd9ecae0-1bdb-4f17-aa84-f8753dbb089b" containerName="curl" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.830935 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9ecae0-1bdb-4f17-aa84-f8753dbb089b" containerName="curl" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831004 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="kube-rbac-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831017 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="method-gate" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831026 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="76090f72-3199-4646-ab80-8809a4284b0f" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831035 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="prom-label-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831045 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="kube-rbac-proxy" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831054 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec664d8b-96f3-4907-83cc-647a27ff7fda" containerName="otc-container" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831064 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9ecae0-1bdb-4f17-aa84-f8753dbb089b" containerName="curl" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831071 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="da85f980-ec6f-4c5a-afc2-b84ed71ac9da" containerName="method-gate" Oct 31 00:28:44.831114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.831080 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfadc60c-8b53-4331-91d7-7b131492017f" containerName="perses" Oct 31 00:28:44.855831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.855801 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:44.856014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.855908 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:44.866466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.866437 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:28:44.866658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.866569 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:28:44.867338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.867320 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lx78f\"" Oct 31 00:28:44.867445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.867370 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:28:44.868037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.868017 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:44.871909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.871889 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:44.880500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.880476 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:44.957941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.957888 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:44.965045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.965023 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:28:44.966612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:44.966592 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:28:45.035571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035538 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdwp\" (UniqueName: \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.035778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035580 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btdj7\" (UniqueName: \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.035778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035604 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.035778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035753 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.035943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035809 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.035943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035856 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.036045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.035956 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.095322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.095237 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.136489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136461 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.136489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136490 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.136646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136510 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.136646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136591 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.136728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136646 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.136728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136692 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdwp\" (UniqueName: \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.136838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136781 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btdj7\" (UniqueName: \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.137002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.136982 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.137266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.137243 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.137375 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.137242 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.139198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.139179 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.139299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.139292 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.150038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.150013 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdwp\" (UniqueName: \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:45.150117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.150079 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btdj7\" (UniqueName: \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\") pod \"data-science-collector-collector-0\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.172989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.172970 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:28:45.449011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.448932 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:28:45.452303 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:28:45.452271 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8757237_ec02_4bd5_9298_579125604407.slice/crio-856fc2ed9560668b7eb02e14915e83ac6050520e09a88d96d72aea7a61a1581d WatchSource:0}: Error finding container 856fc2ed9560668b7eb02e14915e83ac6050520e09a88d96d72aea7a61a1581d: Status 404 returned error can't find the container with id 856fc2ed9560668b7eb02e14915e83ac6050520e09a88d96d72aea7a61a1581d Oct 31 00:28:45.454605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.454275 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:45.454605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.454342 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:28:45.750315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.750281 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8757237-ec02-4bd5-9298-579125604407","Type":"ContainerStarted","Data":"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4"} Oct 31 00:28:45.750734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.750344 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8757237-ec02-4bd5-9298-579125604407","Type":"ContainerStarted","Data":"856fc2ed9560668b7eb02e14915e83ac6050520e09a88d96d72aea7a61a1581d"} Oct 31 00:28:45.773818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.773772 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.7737570969999998 podStartE2EDuration="1.773757097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:45.772947708 +0000 UTC m=+1764.764607319" watchObservedRunningTime="2026-10-31 00:28:45.773757097 +0000 UTC m=+1764.765416683" Oct 31 00:28:45.850215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.850176 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:45.978366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:45.978331 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:28:46.549429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.549395 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.556592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.556570 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.650499 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.650472 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.650637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.650608 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.654187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.654169 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.744861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.744836 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.765330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.765305 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:46.985102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:46.985018 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:47.258636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.258572 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:28:47.261443 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:28:47.261414 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod389e682c_c4cc_40e2_b37f_63fbf82d7873.slice/crio-ffcb1b07caa1afd2bef585ab36cb1678b4f5573a92851ccba67acd23a14d25d7 WatchSource:0}: Error finding container ffcb1b07caa1afd2bef585ab36cb1678b4f5573a92851ccba67acd23a14d25d7: Status 404 returned error can't find the container with id ffcb1b07caa1afd2bef585ab36cb1678b4f5573a92851ccba67acd23a14d25d7 Oct 31 00:28:47.761294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.761259 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"389e682c-c4cc-40e2-b37f-63fbf82d7873","Type":"ContainerStarted","Data":"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21"} Oct 31 00:28:47.761449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.761305 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"389e682c-c4cc-40e2-b37f-63fbf82d7873","Type":"ContainerStarted","Data":"ffcb1b07caa1afd2bef585ab36cb1678b4f5573a92851ccba67acd23a14d25d7"} Oct 31 00:28:47.761449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.761378 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:47.781227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.781165 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=3.781153251 podStartE2EDuration="3.781153251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:28:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.779949802 +0000 UTC m=+1766.771609400" watchObservedRunningTime="2026-10-31 00:28:47.781153251 +0000 UTC m=+1766.772812829" Oct 31 00:28:47.845902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:47.845865 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.43:9093/-/ready\": dial tcp 10.133.0.43:9093: connect: connection refused" Oct 31 00:28:51.509790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.509762 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:28:51.691411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691329 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691363 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691393 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691417 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691589 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691677 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.691914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.691773 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\") pod \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\" (UID: \"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947\") " Oct 31 00:28:51.692217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.692171 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:51.694107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.694078 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:51.694194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.694101 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:51.694296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.694278 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:51.694537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.694521 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "kube-api-access-qdnc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:51.698607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.698542 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:51.705317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.705296 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config" pod "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" (UID: "94b9d0e9-34b0-45d6-a8ba-8f6da9f92947"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:28:51.781390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.781357 5577 generic.go:356] "Generic (PLEG): container finished" podID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerID="96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2" exitCode=0 Oct 31 00:28:51.781503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.781421 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:28:51.781503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.781439 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerDied","Data":"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2"} Oct 31 00:28:51.781503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.781495 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"94b9d0e9-34b0-45d6-a8ba-8f6da9f92947","Type":"ContainerDied","Data":"83b21b9975443902f221d44cb8e09b76e1fe044f3edff619f8f7a378fd7a00f7"} Oct 31 00:28:51.781621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.781521 5577 scope.go:122] "RemoveContainer" containerID="377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8" Oct 31 00:28:51.790166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.790151 5577 scope.go:122] "RemoveContainer" containerID="96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2" Oct 31 00:28:51.792719 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792689 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792728 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792742 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdnc6\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-kube-api-access-qdnc6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792753 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792761 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792769 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.792791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.792776 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:51.797672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.797659 5577 scope.go:122] "RemoveContainer" containerID="34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f" Oct 31 00:28:51.804824 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.804807 5577 scope.go:122] "RemoveContainer" containerID="377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8" Oct 31 00:28:51.805112 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:51.805094 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8\": container with ID starting with 377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8 not found: ID does not exist" containerID="377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8" Oct 31 00:28:51.805179 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.805122 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8"} err="failed to get container status \"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8\": rpc error: code = NotFound desc = could not find container \"377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8\": container with ID starting with 377deebb0d7fb1feecc4ef4ed66031c1c4840d59453e498cf803d57f49499cb8 not found: ID does not exist" Oct 31 00:28:51.805179 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.805140 5577 scope.go:122] "RemoveContainer" containerID="96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2" Oct 31 00:28:51.805394 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:51.805376 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2\": container with ID starting with 96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2 not found: ID does not exist" containerID="96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2" Oct 31 00:28:51.805441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.805401 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2"} err="failed to get container status \"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2\": rpc error: code = NotFound desc = could not find container \"96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2\": container with ID starting with 96f08157dea6e3d511e91879b10c638a0ee90b32f6c93d51fa171b5d7962b1c2 not found: ID does not exist" Oct 31 00:28:51.805441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.805418 5577 scope.go:122] "RemoveContainer" containerID="34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f" Oct 31 00:28:51.805673 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:51.805651 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f\": container with ID starting with 34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f not found: ID does not exist" containerID="34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f" Oct 31 00:28:51.805839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.805685 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f"} err="failed to get container status \"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f\": rpc error: code = NotFound desc = could not find container \"34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f\": container with ID starting with 34e1f0939a7f07287e64d0c0533bd566ecf3937a8ade814fff58ae3154ef722f not found: ID does not exist" Oct 31 00:28:51.807671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.807652 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:28:51.812814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:51.812780 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:28:53.128956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:53.128927 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" path="/var/lib/kubelet/pods/94b9d0e9-34b0-45d6-a8ba-8f6da9f92947/volumes" Oct 31 00:28:58.768539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:28:58.768508 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:28:59.291459 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:28:59.291417 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:29:21.305198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:21.305170 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:29:21.315501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:21.315475 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:29:26.128614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:26.128575 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:29.291964 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:29.291873 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:29:35.105851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.105821 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:35.106387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.106070 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="d8757237-ec02-4bd5-9298-579125604407" containerName="otc-container" containerID="cri-o://ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4" gracePeriod=30 Oct 31 00:29:35.179186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.179154 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:35.217023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.216988 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:35.217269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.217223 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="389e682c-c4cc-40e2-b37f-63fbf82d7873" containerName="perses" containerID="cri-o://98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21" gracePeriod=30 Oct 31 00:29:35.312089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.312056 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:35.394766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.394741 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:35.464453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.464431 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:35.517075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517050 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\" (UniqueName: \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\") pod \"389e682c-c4cc-40e2-b37f-63fbf82d7873\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " Oct 31 00:29:35.517229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517100 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\") pod \"d8757237-ec02-4bd5-9298-579125604407\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " Oct 31 00:29:35.517229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517131 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\") pod \"d8757237-ec02-4bd5-9298-579125604407\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " Oct 31 00:29:35.517229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517150 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\") pod \"d8757237-ec02-4bd5-9298-579125604407\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " Oct 31 00:29:35.517410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517247 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"389e682c-c4cc-40e2-b37f-63fbf82d7873\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " Oct 31 00:29:35.517410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517271 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\" (UniqueName: \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\") pod \"d8757237-ec02-4bd5-9298-579125604407\" (UID: \"d8757237-ec02-4bd5-9298-579125604407\") " Oct 31 00:29:35.517410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517307 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\" (UniqueName: \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\") pod \"389e682c-c4cc-40e2-b37f-63fbf82d7873\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " Oct 31 00:29:35.517410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517329 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\" (UniqueName: \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\") pod \"389e682c-c4cc-40e2-b37f-63fbf82d7873\" (UID: \"389e682c-c4cc-40e2-b37f-63fbf82d7873\") " Oct 31 00:29:35.517621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517592 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal" pod "d8757237-ec02-4bd5-9298-579125604407" (UID: "d8757237-ec02-4bd5-9298-579125604407"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:35.518109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.517808 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config" pod "389e682c-c4cc-40e2-b37f-63fbf82d7873" (UID: "389e682c-c4cc-40e2-b37f-63fbf82d7873"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:35.519579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.519558 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs" pod "d8757237-ec02-4bd5-9298-579125604407" (UID: "d8757237-ec02-4bd5-9298-579125604407"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.520095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.520070 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs" pod "d8757237-ec02-4bd5-9298-579125604407" (UID: "d8757237-ec02-4bd5-9298-579125604407"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:35.520095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.520079 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7" pod "d8757237-ec02-4bd5-9298-579125604407" (UID: "d8757237-ec02-4bd5-9298-579125604407"). InnerVolumeSpecName "kube-api-access-btdj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:35.520441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.520421 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp" pod "389e682c-c4cc-40e2-b37f-63fbf82d7873" (UID: "389e682c-c4cc-40e2-b37f-63fbf82d7873"). InnerVolumeSpecName "kube-api-access-czdwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:35.521274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.521252 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "389e682c-c4cc-40e2-b37f-63fbf82d7873" (UID: "389e682c-c4cc-40e2-b37f-63fbf82d7873"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:29:35.618205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618169 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:29:35.618205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618201 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btdj7\" (UniqueName: \"kubernetes.io/projected/d8757237-ec02-4bd5-9298-579125604407-kube-api-access-btdj7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.618409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618217 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdwp\" (UniqueName: \"kubernetes.io/projected/389e682c-c4cc-40e2-b37f-63fbf82d7873-kube-api-access-czdwp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.618409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618230 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/389e682c-c4cc-40e2-b37f-63fbf82d7873-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.618409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618244 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.618409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618254 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d8757237-ec02-4bd5-9298-579125604407-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.618409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.618266 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d8757237-ec02-4bd5-9298-579125604407-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.631549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.631531 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:29:35.664679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.664652 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins" pod "389e682c-c4cc-40e2-b37f-63fbf82d7873" (UID: "389e682c-c4cc-40e2-b37f-63fbf82d7873"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:35.719176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.719152 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:29:35.719176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.719174 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/389e682c-c4cc-40e2-b37f-63fbf82d7873-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:35.972854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.972751 5577 generic.go:356] "Generic (PLEG): container finished" podID="d8757237-ec02-4bd5-9298-579125604407" containerID="ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4" exitCode=0 Oct 31 00:29:35.973010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.972918 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:35.973082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.972903 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8757237-ec02-4bd5-9298-579125604407","Type":"ContainerDied","Data":"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4"} Oct 31 00:29:35.973143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.973124 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"d8757237-ec02-4bd5-9298-579125604407","Type":"ContainerDied","Data":"856fc2ed9560668b7eb02e14915e83ac6050520e09a88d96d72aea7a61a1581d"} Oct 31 00:29:35.973203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.973164 5577 scope.go:122] "RemoveContainer" containerID="ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4" Oct 31 00:29:35.974371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.974217 5577 generic.go:356] "Generic (PLEG): container finished" podID="389e682c-c4cc-40e2-b37f-63fbf82d7873" containerID="98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21" exitCode=0 Oct 31 00:29:35.974371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.974261 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:35.974371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.974264 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"389e682c-c4cc-40e2-b37f-63fbf82d7873","Type":"ContainerDied","Data":"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21"} Oct 31 00:29:35.974371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.974295 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"389e682c-c4cc-40e2-b37f-63fbf82d7873","Type":"ContainerDied","Data":"ffcb1b07caa1afd2bef585ab36cb1678b4f5573a92851ccba67acd23a14d25d7"} Oct 31 00:29:35.982413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.982397 5577 scope.go:122] "RemoveContainer" containerID="ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4" Oct 31 00:29:35.982695 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:35.982671 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4\": container with ID starting with ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4 not found: ID does not exist" containerID="ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4" Oct 31 00:29:35.982803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.982699 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4"} err="failed to get container status \"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4\": rpc error: code = NotFound desc = could not find container \"ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4\": container with ID starting with ad684387f73596b0135db252c16484f22d2114da08dd9dcbfddfee00190fe2d4 not found: ID does not exist" Oct 31 00:29:35.982803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.982747 5577 scope.go:122] "RemoveContainer" containerID="98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21" Oct 31 00:29:35.990216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.990202 5577 scope.go:122] "RemoveContainer" containerID="98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21" Oct 31 00:29:35.990445 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:35.990430 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21\": container with ID starting with 98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21 not found: ID does not exist" containerID="98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21" Oct 31 00:29:35.990495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.990451 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21"} err="failed to get container status \"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21\": rpc error: code = NotFound desc = could not find container \"98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21\": container with ID starting with 98d218679bf32de952775fa17097573873140ee9fb380005bd469a0a1663ec21 not found: ID does not exist" Oct 31 00:29:35.995810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:35.995791 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:36.002150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:36.002129 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:36.013488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:36.013468 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:36.017868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:36.017851 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:37.128630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:37.128598 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="389e682c-c4cc-40e2-b37f-63fbf82d7873" path="/var/lib/kubelet/pods/389e682c-c4cc-40e2-b37f-63fbf82d7873/volumes" Oct 31 00:29:37.129031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:37.128985 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8757237-ec02-4bd5-9298-579125604407" path="/var/lib/kubelet/pods/d8757237-ec02-4bd5-9298-579125604407/volumes" Oct 31 00:29:37.979395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:37.979347 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.104527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.104489 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.164771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.164742 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165047 5577 cpu_manager.go:414] "containerMap: removing container" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="init-config-reloader" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165059 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="init-config-reloader" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165084 5577 cpu_manager.go:414] "containerMap: removing container" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165089 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165099 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d8757237-ec02-4bd5-9298-579125604407" containerName="otc-container" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8757237-ec02-4bd5-9298-579125604407" containerName="otc-container" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165113 5577 cpu_manager.go:414] "containerMap: removing container" podUID="389e682c-c4cc-40e2-b37f-63fbf82d7873" containerName="perses" Oct 31 00:29:38.165124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165120 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="389e682c-c4cc-40e2-b37f-63fbf82d7873" containerName="perses" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165130 5577 cpu_manager.go:414] "containerMap: removing container" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="config-reloader" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165135 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="config-reloader" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165182 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="config-reloader" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165198 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b9d0e9-34b0-45d6-a8ba-8f6da9f92947" containerName="alertmanager" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165205 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8757237-ec02-4bd5-9298-579125604407" containerName="otc-container" Oct 31 00:29:38.165359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.165210 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="389e682c-c4cc-40e2-b37f-63fbf82d7873" containerName="perses" Oct 31 00:29:38.191786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.191752 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:38.191960 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.191846 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.206867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.206838 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:38.209998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.209974 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.222032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.222008 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:38.239578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.239525 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4925x\" (UniqueName: \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.239578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.239554 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.239741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.239583 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.239741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.239655 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.254442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.254418 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.266935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.266913 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:29:38.266935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.266920 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:29:38.267081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.266951 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:29:38.292432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.292403 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:38.292535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.292502 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:29:38.292535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.292503 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:29:38.292535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.292527 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:38.340956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.340937 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.341061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.340966 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.341061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341002 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4925x\" (UniqueName: \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.341183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341169 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.341221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341208 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.341275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341226 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzgxs\" (UniqueName: \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.341275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341246 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.341275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341269 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.341895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.341875 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.343675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.343656 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.343787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.343693 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.347863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.347843 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.353296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.353273 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4925x\" (UniqueName: \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\") pod \"data-science-collector-collector-0\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.382127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.382100 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.442229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442202 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzgxs\" (UniqueName: \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442239 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442261 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442282 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442362 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442602 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442585 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.442855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.442839 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.445828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.445807 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.455173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.455143 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzgxs\" (UniqueName: \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.506640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.506616 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:38.509127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.509102 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:38.539247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.539218 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.559334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.559312 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.794519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.794442 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:38.797500 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:29:38.797471 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7e7092_b782_44e2_bed9_e65ae0a8ad50.slice/crio-736080c5bc329c14239d090421f523cba3c1638997a86ec714604e8ec8179048 WatchSource:0}: Error finding container 736080c5bc329c14239d090421f523cba3c1638997a86ec714604e8ec8179048: Status 404 returned error can't find the container with id 736080c5bc329c14239d090421f523cba3c1638997a86ec714604e8ec8179048 Oct 31 00:29:38.799500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.799470 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:38.799561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.799531 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:38.821782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.821749 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:38.994084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.994044 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f7e7092-b782-44e2-bed9-e65ae0a8ad50","Type":"ContainerStarted","Data":"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404"} Oct 31 00:29:38.994271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:38.994123 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f7e7092-b782-44e2-bed9-e65ae0a8ad50","Type":"ContainerStarted","Data":"736080c5bc329c14239d090421f523cba3c1638997a86ec714604e8ec8179048"} Oct 31 00:29:39.014975 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:39.014926 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.014913137 podStartE2EDuration="1.014913137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:39.014564867 +0000 UTC m=+1818.006224454" watchObservedRunningTime="2026-10-31 00:29:39.014913137 +0000 UTC m=+1818.006572711" Oct 31 00:29:39.115481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:39.115412 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:39.118557 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:29:39.118520 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3713262f_4296_4ed5_89ce_45a82a619760.slice/crio-c11671190056f489508c6986781de996da424d11573d4a28e7fcf5e1385ed6d9 WatchSource:0}: Error finding container c11671190056f489508c6986781de996da424d11573d4a28e7fcf5e1385ed6d9: Status 404 returned error can't find the container with id c11671190056f489508c6986781de996da424d11573d4a28e7fcf5e1385ed6d9 Oct 31 00:29:39.510851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:39.510818 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:39.638451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:39.638417 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:40.000631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:40.000594 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"3713262f-4296-4ed5-89ce-45a82a619760","Type":"ContainerStarted","Data":"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3"} Oct 31 00:29:40.000814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:40.000643 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"3713262f-4296-4ed5-89ce-45a82a619760","Type":"ContainerStarted","Data":"c11671190056f489508c6986781de996da424d11573d4a28e7fcf5e1385ed6d9"} Oct 31 00:29:40.000906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:40.000883 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:40.020521 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:40.020476 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=2.020463635 podStartE2EDuration="2.020463635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:40.019151255 +0000 UTC m=+1819.010810842" watchObservedRunningTime="2026-10-31 00:29:40.020463635 +0000 UTC m=+1819.012123262" Oct 31 00:29:51.007865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:51.007835 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:52.031669 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.031641 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:52.032194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.031878 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" containerName="otc-container" containerID="cri-o://468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404" gracePeriod=30 Oct 31 00:29:52.120435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.120024 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:52.135691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.135661 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:52.135951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.135918 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="3713262f-4296-4ed5-89ce-45a82a619760" containerName="perses" containerID="cri-o://13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3" gracePeriod=30 Oct 31 00:29:52.305419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.305395 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:52.347529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.347479 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\") pod \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " Oct 31 00:29:52.347529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.347530 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\") pod \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " Oct 31 00:29:52.347781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.347582 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\" (UniqueName: \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\") pod \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " Oct 31 00:29:52.347781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.347600 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\") pod \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\" (UID: \"7f7e7092-b782-44e2-bed9-e65ae0a8ad50\") " Oct 31 00:29:52.347983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.347958 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal" pod "7f7e7092-b782-44e2-bed9-e65ae0a8ad50" (UID: "7f7e7092-b782-44e2-bed9-e65ae0a8ad50"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.349788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.349763 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs" pod "7f7e7092-b782-44e2-bed9-e65ae0a8ad50" (UID: "7f7e7092-b782-44e2-bed9-e65ae0a8ad50"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.349887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.349785 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x" pod "7f7e7092-b782-44e2-bed9-e65ae0a8ad50" (UID: "7f7e7092-b782-44e2-bed9-e65ae0a8ad50"). InnerVolumeSpecName "kube-api-access-4925x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.349887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.349857 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs" pod "7f7e7092-b782-44e2-bed9-e65ae0a8ad50" (UID: "7f7e7092-b782-44e2-bed9-e65ae0a8ad50"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:52.390825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.390806 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:52.448005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.447971 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\" (UniqueName: \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\") pod \"3713262f-4296-4ed5-89ce-45a82a619760\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " Oct 31 00:29:52.448142 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448011 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\" (UniqueName: \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\") pod \"3713262f-4296-4ed5-89ce-45a82a619760\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " Oct 31 00:29:52.448142 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448064 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\" (UniqueName: \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\") pod \"3713262f-4296-4ed5-89ce-45a82a619760\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " Oct 31 00:29:52.448230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448180 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"3713262f-4296-4ed5-89ce-45a82a619760\" (UID: \"3713262f-4296-4ed5-89ce-45a82a619760\") " Oct 31 00:29:52.448451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448430 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4925x\" (UniqueName: \"kubernetes.io/projected/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-kube-api-access-4925x\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.448537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448465 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.448537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448482 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.448537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448496 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7f7e7092-b782-44e2-bed9-e65ae0a8ad50-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.448537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.448432 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config" pod "3713262f-4296-4ed5-89ce-45a82a619760" (UID: "3713262f-4296-4ed5-89ce-45a82a619760"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:52.450434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.450398 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs" pod "3713262f-4296-4ed5-89ce-45a82a619760" (UID: "3713262f-4296-4ed5-89ce-45a82a619760"). InnerVolumeSpecName "kube-api-access-gzgxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:52.451417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.451380 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "3713262f-4296-4ed5-89ce-45a82a619760" (UID: "3713262f-4296-4ed5-89ce-45a82a619760"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:29:52.549603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.549529 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzgxs\" (UniqueName: \"kubernetes.io/projected/3713262f-4296-4ed5-89ce-45a82a619760-kube-api-access-gzgxs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.549603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.549574 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:29:52.549603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.549585 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3713262f-4296-4ed5-89ce-45a82a619760-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.562694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.562676 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:29:52.599959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.599916 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins" pod "3713262f-4296-4ed5-89ce-45a82a619760" (UID: "3713262f-4296-4ed5-89ce-45a82a619760"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:52.650325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.650295 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/3713262f-4296-4ed5-89ce-45a82a619760-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:52.650325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:52.650324 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:29:53.052891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.052857 5577 generic.go:356] "Generic (PLEG): container finished" podID="3713262f-4296-4ed5-89ce-45a82a619760" containerID="13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3" exitCode=0 Oct 31 00:29:53.053326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.052919 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:53.053326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.052941 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"3713262f-4296-4ed5-89ce-45a82a619760","Type":"ContainerDied","Data":"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3"} Oct 31 00:29:53.053326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.052995 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"3713262f-4296-4ed5-89ce-45a82a619760","Type":"ContainerDied","Data":"c11671190056f489508c6986781de996da424d11573d4a28e7fcf5e1385ed6d9"} Oct 31 00:29:53.053326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.053019 5577 scope.go:122] "RemoveContainer" containerID="13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3" Oct 31 00:29:53.054381 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.054352 5577 generic.go:356] "Generic (PLEG): container finished" podID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" containerID="468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404" exitCode=0 Oct 31 00:29:53.054479 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.054432 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:53.054688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.054660 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f7e7092-b782-44e2-bed9-e65ae0a8ad50","Type":"ContainerDied","Data":"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404"} Oct 31 00:29:53.054820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.054769 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"7f7e7092-b782-44e2-bed9-e65ae0a8ad50","Type":"ContainerDied","Data":"736080c5bc329c14239d090421f523cba3c1638997a86ec714604e8ec8179048"} Oct 31 00:29:53.061156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.061129 5577 scope.go:122] "RemoveContainer" containerID="13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3" Oct 31 00:29:53.061424 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:53.061402 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3\": container with ID starting with 13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3 not found: ID does not exist" containerID="13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3" Oct 31 00:29:53.061517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.061437 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3"} err="failed to get container status \"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3\": rpc error: code = NotFound desc = could not find container \"13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3\": container with ID starting with 13d5dcd49197aacaf7e0e017e5aec2001a548b15e646bc687e62a72add6578a3 not found: ID does not exist" Oct 31 00:29:53.061517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.061458 5577 scope.go:122] "RemoveContainer" containerID="468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404" Oct 31 00:29:53.074479 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.074461 5577 scope.go:122] "RemoveContainer" containerID="468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404" Oct 31 00:29:53.074778 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:53.074753 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404\": container with ID starting with 468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404 not found: ID does not exist" containerID="468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404" Oct 31 00:29:53.074849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.074786 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404"} err="failed to get container status \"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404\": rpc error: code = NotFound desc = could not find container \"468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404\": container with ID starting with 468cd9baad6a213481ffc919c9cce88625663c4c835cf73a23d52455c2321404 not found: ID does not exist" Oct 31 00:29:53.086422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.086402 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:53.092530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.092502 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:53.103206 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.103185 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:53.106904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.106886 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:53.129315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.129291 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3713262f-4296-4ed5-89ce-45a82a619760" path="/var/lib/kubelet/pods/3713262f-4296-4ed5-89ce-45a82a619760/volumes" Oct 31 00:29:53.129643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.129631 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" path="/var/lib/kubelet/pods/7f7e7092-b782-44e2-bed9-e65ae0a8ad50/volumes" Oct 31 00:29:53.914593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:53.914561 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.746462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.746429 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.868071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.868041 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:54.949061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949029 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:54.949365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949352 5577 cpu_manager.go:414] "containerMap: removing container" podUID="3713262f-4296-4ed5-89ce-45a82a619760" containerName="perses" Oct 31 00:29:54.949418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949366 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3713262f-4296-4ed5-89ce-45a82a619760" containerName="perses" Oct 31 00:29:54.949418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949397 5577 cpu_manager.go:414] "containerMap: removing container" podUID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" containerName="otc-container" Oct 31 00:29:54.949418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949402 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" containerName="otc-container" Oct 31 00:29:54.949526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949453 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f7e7092-b782-44e2-bed9-e65ae0a8ad50" containerName="otc-container" Oct 31 00:29:54.949526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.949462 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="3713262f-4296-4ed5-89ce-45a82a619760" containerName="perses" Oct 31 00:29:54.954154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.954129 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:54.962405 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.962324 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:29:54.962575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.962562 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:29:54.963500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.963477 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:29:54.963625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.963560 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:54.963625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.963607 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:29:54.963877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.963766 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:29:54.985850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.985827 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:54.989353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.989336 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:54.993629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.992770 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:29:54.998744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:54.998697 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:55.012204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.012179 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:55.062988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.062962 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:29:55.068495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068472 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.068589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068509 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.068589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068540 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjmpq\" (UniqueName: \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.068687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068598 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp66k\" (UniqueName: \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.068687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068623 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.068687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068648 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.068794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.068697 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.139075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.139038 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:55.169313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169284 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.169412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169315 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.169412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169334 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.169412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169384 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.169516 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169423 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.169573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169549 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjmpq\" (UniqueName: \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.169839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169619 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp66k\" (UniqueName: \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.169839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.169775 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.170075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.170060 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.170429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.170408 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.171959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.171937 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.171959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.171955 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.181890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.181856 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjmpq\" (UniqueName: \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\") pod \"data-science-collector-collector-0\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.181979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.181924 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp66k\" (UniqueName: \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:55.263522 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.263463 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:55.268440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.268418 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:29:55.513455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.513416 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:55.546117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.546043 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:29:55.549339 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:29:55.549312 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe45d51e_6e39_4505_bb6f_ccfe21bd3c90.slice/crio-6239892f525953e89b1d339836ed7c6c8ecca12f7f9aff2c9aa0bc0a2a51bc96 WatchSource:0}: Error finding container 6239892f525953e89b1d339836ed7c6c8ecca12f7f9aff2c9aa0bc0a2a51bc96: Status 404 returned error can't find the container with id 6239892f525953e89b1d339836ed7c6c8ecca12f7f9aff2c9aa0bc0a2a51bc96 Oct 31 00:29:55.551854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.551823 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:55.551975 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.551887 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:29:55.721995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:55.721957 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:29:56.070728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:56.070675 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"be45d51e-6e39-4505-bb6f-ccfe21bd3c90","Type":"ContainerStarted","Data":"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c"} Oct 31 00:29:56.071218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:56.070763 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"be45d51e-6e39-4505-bb6f-ccfe21bd3c90","Type":"ContainerStarted","Data":"6239892f525953e89b1d339836ed7c6c8ecca12f7f9aff2c9aa0bc0a2a51bc96"} Oct 31 00:29:56.089835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:56.089783 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.089767101 podStartE2EDuration="2.089767101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:56.088977581 +0000 UTC m=+1835.080637161" watchObservedRunningTime="2026-10-31 00:29:56.089767101 +0000 UTC m=+1835.081426683" Oct 31 00:29:58.897670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:58.897621 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:58.904561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:58.904533 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:58.999008 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:58.998984 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:58.999110 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:58.999071 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:59.002463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:59.002448 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:59.093393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:59.093368 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:59.111438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:59.111414 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:59.205478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:59.205391 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:29:59.293583 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:29:59.293537 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:29:59.477921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:29:59.477851 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:29:59.481159 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:29:59.481130 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0abeb12e_0052_4afe_ad5f_b0c099cc2398.slice/crio-872d3028457fdf0892ab3b747ebe3acb5ec3ca928f0cc3bc5bb042bec4abf744 WatchSource:0}: Error finding container 872d3028457fdf0892ab3b747ebe3acb5ec3ca928f0cc3bc5bb042bec4abf744: Status 404 returned error can't find the container with id 872d3028457fdf0892ab3b747ebe3acb5ec3ca928f0cc3bc5bb042bec4abf744 Oct 31 00:30:00.089603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:00.089571 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0abeb12e-0052-4afe-ad5f-b0c099cc2398","Type":"ContainerStarted","Data":"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377"} Oct 31 00:30:00.089988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:00.089610 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0abeb12e-0052-4afe-ad5f-b0c099cc2398","Type":"ContainerStarted","Data":"872d3028457fdf0892ab3b747ebe3acb5ec3ca928f0cc3bc5bb042bec4abf744"} Oct 31 00:30:00.089988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:00.089633 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:30:00.109612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:00.109561 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.109548589 podStartE2EDuration="6.109548589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:00.108260444 +0000 UTC m=+1839.099920038" watchObservedRunningTime="2026-10-31 00:30:00.109548589 +0000 UTC m=+1839.101208166" Oct 31 00:30:11.096689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:11.096662 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:30:26.185279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:26.185235 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:29.292945 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:30:29.292904 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:30:47.251486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.251448 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:30:47.252050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.251667 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" containerName="otc-container" containerID="cri-o://98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c" gracePeriod=30 Oct 31 00:30:47.314247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.314207 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.371738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.371633 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:30:47.371920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.371866 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" containerName="perses" containerID="cri-o://457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377" gracePeriod=30 Oct 31 00:30:47.453261 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.453226 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.562158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.562127 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:30:47.587362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.587327 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:47.623201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.623178 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:30:47.710989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.710953 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\" (UniqueName: \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\") pod \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " Oct 31 00:30:47.711169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.711001 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\") pod \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " Oct 31 00:30:47.711169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.711036 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\" (UniqueName: \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\") pod \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " Oct 31 00:30:47.711169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.711060 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\") pod \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\" (UID: \"be45d51e-6e39-4505-bb6f-ccfe21bd3c90\") " Oct 31 00:30:47.711434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.711397 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal" pod "be45d51e-6e39-4505-bb6f-ccfe21bd3c90" (UID: "be45d51e-6e39-4505-bb6f-ccfe21bd3c90"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:47.713392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.713370 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs" pod "be45d51e-6e39-4505-bb6f-ccfe21bd3c90" (UID: "be45d51e-6e39-4505-bb6f-ccfe21bd3c90"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.713493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.713434 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs" pod "be45d51e-6e39-4505-bb6f-ccfe21bd3c90" (UID: "be45d51e-6e39-4505-bb6f-ccfe21bd3c90"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:47.713493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.713457 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq" pod "be45d51e-6e39-4505-bb6f-ccfe21bd3c90" (UID: "be45d51e-6e39-4505-bb6f-ccfe21bd3c90"). InnerVolumeSpecName "kube-api-access-fjmpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:47.811969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.811891 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\" (UniqueName: \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\") pod \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " Oct 31 00:30:47.812104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812047 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " Oct 31 00:30:47.812104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812092 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\" (UniqueName: \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\") pod \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " Oct 31 00:30:47.812193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812119 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\" (UniqueName: \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\") pod \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\" (UID: \"0abeb12e-0052-4afe-ad5f-b0c099cc2398\") " Oct 31 00:30:47.812253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812242 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.812315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812255 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjmpq\" (UniqueName: \"kubernetes.io/projected/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-kube-api-access-fjmpq\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.812315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812264 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.812315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812274 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/be45d51e-6e39-4505-bb6f-ccfe21bd3c90-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.812579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.812547 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config" pod "0abeb12e-0052-4afe-ad5f-b0c099cc2398" (UID: "0abeb12e-0052-4afe-ad5f-b0c099cc2398"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:47.814457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.814425 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k" pod "0abeb12e-0052-4afe-ad5f-b0c099cc2398" (UID: "0abeb12e-0052-4afe-ad5f-b0c099cc2398"). InnerVolumeSpecName "kube-api-access-hp66k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:47.815061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.815041 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "0abeb12e-0052-4afe-ad5f-b0c099cc2398" (UID: "0abeb12e-0052-4afe-ad5f-b0c099cc2398"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:30:47.913496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.913463 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp66k\" (UniqueName: \"kubernetes.io/projected/0abeb12e-0052-4afe-ad5f-b0c099cc2398-kube-api-access-hp66k\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.913496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.913491 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0abeb12e-0052-4afe-ad5f-b0c099cc2398-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.913687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.913521 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:30:47.926340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.926321 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:30:47.957048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:47.957017 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins" pod "0abeb12e-0052-4afe-ad5f-b0c099cc2398" (UID: "0abeb12e-0052-4afe-ad5f-b0c099cc2398"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:48.014644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.014615 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/0abeb12e-0052-4afe-ad5f-b0c099cc2398-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:48.014644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.014642 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:30:48.285680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.285648 5577 generic.go:356] "Generic (PLEG): container finished" podID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" containerID="457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377" exitCode=0 Oct 31 00:30:48.286076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.285688 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0abeb12e-0052-4afe-ad5f-b0c099cc2398","Type":"ContainerDied","Data":"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377"} Oct 31 00:30:48.286076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.285695 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:30:48.286076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.285745 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"0abeb12e-0052-4afe-ad5f-b0c099cc2398","Type":"ContainerDied","Data":"872d3028457fdf0892ab3b747ebe3acb5ec3ca928f0cc3bc5bb042bec4abf744"} Oct 31 00:30:48.286076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.285761 5577 scope.go:122] "RemoveContainer" containerID="457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377" Oct 31 00:30:48.287254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.287228 5577 generic.go:356] "Generic (PLEG): container finished" podID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" containerID="98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c" exitCode=0 Oct 31 00:30:48.287746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.287269 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:30:48.287746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.287259 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"be45d51e-6e39-4505-bb6f-ccfe21bd3c90","Type":"ContainerDied","Data":"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c"} Oct 31 00:30:48.287746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.287379 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"be45d51e-6e39-4505-bb6f-ccfe21bd3c90","Type":"ContainerDied","Data":"6239892f525953e89b1d339836ed7c6c8ecca12f7f9aff2c9aa0bc0a2a51bc96"} Oct 31 00:30:48.294804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.294781 5577 scope.go:122] "RemoveContainer" containerID="457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377" Oct 31 00:30:48.295091 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:30:48.295071 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377\": container with ID starting with 457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377 not found: ID does not exist" containerID="457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377" Oct 31 00:30:48.295146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.295101 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377"} err="failed to get container status \"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377\": rpc error: code = NotFound desc = could not find container \"457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377\": container with ID starting with 457e7591e92e10bbf5019cc97186a7190a5264a656a512a9770f85bd834a0377 not found: ID does not exist" Oct 31 00:30:48.295146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.295116 5577 scope.go:122] "RemoveContainer" containerID="98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c" Oct 31 00:30:48.302473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.302457 5577 scope.go:122] "RemoveContainer" containerID="98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c" Oct 31 00:30:48.302721 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:30:48.302689 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c\": container with ID starting with 98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c not found: ID does not exist" containerID="98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c" Oct 31 00:30:48.302790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.302743 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c"} err="failed to get container status \"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c\": rpc error: code = NotFound desc = could not find container \"98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c\": container with ID starting with 98d03cf78bdcc89c315cc12644e369e1250856000f618dc5b5b8f3578cd58f7c not found: ID does not exist" Oct 31 00:30:48.309215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.309196 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:30:48.313583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.313560 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:30:48.324823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.324806 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:30:48.329254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:48.329236 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:30:49.129397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:49.129364 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" path="/var/lib/kubelet/pods/0abeb12e-0052-4afe-ad5f-b0c099cc2398/volumes" Oct 31 00:30:49.129759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:49.129746 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" path="/var/lib/kubelet/pods/be45d51e-6e39-4505-bb6f-ccfe21bd3c90/volumes" Oct 31 00:30:51.609101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609073 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609354 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" containerName="perses" Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609364 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" containerName="perses" Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609380 5577 cpu_manager.go:414] "containerMap: removing container" podUID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" containerName="otc-container" Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609385 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" containerName="otc-container" Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609425 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0abeb12e-0052-4afe-ad5f-b0c099cc2398" containerName="perses" Oct 31 00:30:51.609466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.609434 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="be45d51e-6e39-4505-bb6f-ccfe21bd3c90" containerName="otc-container" Oct 31 00:30:51.633733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.633682 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:30:51.633904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.633846 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.638404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.638381 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:30:51.639045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.639022 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:30:51.639045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.639042 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:30:51.644538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.644514 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhrfb\" (UniqueName: \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.644629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.644566 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.719768 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.719735 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:30:51.745791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.745764 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.745914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.745822 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhrfb\" (UniqueName: \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.746132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.746113 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.758698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.758672 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhrfb\" (UniqueName: \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\") pod \"seaweedfs\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:51.946578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:51.946486 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:30:52.215605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:52.215533 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:30:52.218808 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:30:52.218779 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0bba5645_155b_4d63_957a_ba3489aefcd0.slice/crio-1150f04aa9db56229032fa35a7e20ea64ddddf4ad4555be3a2129ef62a77633d WatchSource:0}: Error finding container 1150f04aa9db56229032fa35a7e20ea64ddddf4ad4555be3a2129ef62a77633d: Status 404 returned error can't find the container with id 1150f04aa9db56229032fa35a7e20ea64ddddf4ad4555be3a2129ef62a77633d Oct 31 00:30:52.220570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:52.220551 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:30:52.305537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:30:52.305500 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"0bba5645-155b-4d63-957a-ba3489aefcd0","Type":"ContainerStarted","Data":"1150f04aa9db56229032fa35a7e20ea64ddddf4ad4555be3a2129ef62a77633d"} Oct 31 00:30:59.295963 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:30:59.295861 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:31:02.353095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:02.353054 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"0bba5645-155b-4d63-957a-ba3489aefcd0","Type":"ContainerStarted","Data":"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d"} Oct 31 00:31:02.353486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:02.353187 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:31:02.370505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:02.370453 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/seaweedfs" podStartSLOduration=1.943851681 podStartE2EDuration="11.370438037s" totalImagesPullingTime="9.426586356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:02.370316272 +0000 UTC m=+1901.361975867" watchObservedRunningTime="2026-10-31 00:31:02.370438037 +0000 UTC m=+1901.362097620" Oct 31 00:31:12.116002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:12.115952 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:12.239231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:12.239168 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:13.359890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:13.359854 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:31:21.482954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:21.482918 5577 scope.go:122] "RemoveContainer" containerID="14dd9b704f3751e92bbc1155d0dac04d5348d85518f293086646b72302fc2a15" Oct 31 00:31:26.240911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:26.240868 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:26.878338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:26.878297 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.014220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.014176 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.151564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.151482 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:31:27.154844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.154820 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.158833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.158804 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.164771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.164748 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:31:27.199748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.199689 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.199917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.199816 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbkvh\" (UniqueName: \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.199917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.199855 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.229764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.229699 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:31:27.285166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.285133 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:27.301222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.301195 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.301315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.301269 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.301356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.301312 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbkvh\" (UniqueName: \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.301674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.301657 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.301933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.301914 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.313506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.313487 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbkvh\" (UniqueName: \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:27.415987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:27.415888 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:29.293879 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:31:29.293844 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:31:29.957010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:29.956978 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:29.993963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:29.993927 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:31:29.996829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:29.996733 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.004622 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.004598 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:31:30.004846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.004762 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 31 00:31:30.008025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.008000 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:31:30.019311 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.019289 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"s3-secret\"" Oct 31 00:31:30.019447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.019334 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 31 00:31:30.019447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.019428 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-5wl9c\"" Oct 31 00:31:30.023796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.023763 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.023898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.023809 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvqb2\" (UniqueName: \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.023898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.023884 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.024000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.023981 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.024081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.024060 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.027096 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.027074 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:31:30.030361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.030341 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.041192 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.041166 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:31:30.090588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.090556 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:30.093260 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.093235 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 31 00:31:30.124743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.124698 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.124843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.124752 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.124843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.124784 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.124843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.124814 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99v4l\" (UniqueName: \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.124993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.124866 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.125071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125048 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.125132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125116 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.125188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125155 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvqb2\" (UniqueName: \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.125240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125184 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.125240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125194 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.125336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125264 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.125654 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125638 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.125753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.125643 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.127343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.127326 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.140570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.140543 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvqb2\" (UniqueName: \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\") pod \"tempo-data-science-tempostack-querier-5b7b64f477-5r6wb\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.224617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.224536 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:30.226436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226412 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.226548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226448 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.226548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226478 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.226548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226528 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99v4l\" (UniqueName: \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.226699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226576 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.226998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.226971 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.227217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.227194 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.227361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.227341 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.229696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.229671 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.239279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.239253 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99v4l\" (UniqueName: \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\") pod \"tempo-data-science-tempostack-compactor-7898846dc-64sws\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.248675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.248645 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:30.251780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.251761 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.256126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256035 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:31:30.256126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256075 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 31 00:31:30.256126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256079 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-h9nc8\"" Oct 31 00:31:30.256126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256106 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:31:30.256126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256109 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 31 00:31:30.256491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.256165 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 31 00:31:30.266153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.266120 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:30.312400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.312364 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:30.327656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327630 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.327832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327664 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.327832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327689 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.327832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327769 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.327970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327934 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4r7z\" (UniqueName: \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.328011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.327994 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.328073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.328057 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.345647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.345612 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:30.355933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.355896 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:30.429366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429336 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429378 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429450 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4r7z\" (UniqueName: \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429478 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429532 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429565 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.429665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.429586 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.430293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.430208 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.430605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.430581 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.430767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.430742 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.432286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.432259 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.432779 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.432755 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.434317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.434293 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.445379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.445350 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4r7z\" (UniqueName: \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\") pod \"tempo-data-science-tempostack-gateway-849f997544-j6lbs\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.566526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.566487 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:30.586343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.586218 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:31:30.589761 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:31:30.589726 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00de023_dcb2_4a2a_baf4_396b1897857a.slice/crio-7d51e91cdacf8c9dfa2e1a3a59a36207b31b4cff52f0baa1037dc1166b251478 WatchSource:0}: Error finding container 7d51e91cdacf8c9dfa2e1a3a59a36207b31b4cff52f0baa1037dc1166b251478: Status 404 returned error can't find the container with id 7d51e91cdacf8c9dfa2e1a3a59a36207b31b4cff52f0baa1037dc1166b251478 Oct 31 00:31:30.627920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.627829 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:31:30.633344 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:31:30.633316 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40cae269_904f_4533_baff_d7af79d3e111.slice/crio-ca33108a5b586cba7208e315d86432a99665c5f59253188ef07f928e708a8168 WatchSource:0}: Error finding container ca33108a5b586cba7208e315d86432a99665c5f59253188ef07f928e708a8168: Status 404 returned error can't find the container with id ca33108a5b586cba7208e315d86432a99665c5f59253188ef07f928e708a8168 Oct 31 00:31:30.846046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:30.846013 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:30.849261 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:31:30.849234 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf89856e3_8a50_40b0_a224_e86e999bf6d6.slice/crio-27cb9c05f1ad7ff548a73479989c98939da0bef97ca32ad854fd1dd3251d9047 WatchSource:0}: Error finding container 27cb9c05f1ad7ff548a73479989c98939da0bef97ca32ad854fd1dd3251d9047: Status 404 returned error can't find the container with id 27cb9c05f1ad7ff548a73479989c98939da0bef97ca32ad854fd1dd3251d9047 Oct 31 00:31:31.136315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.136221 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.142966 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.142942 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.242993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.242953 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.243344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.243320 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.246396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.246373 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.337303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.337262 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.357897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.357870 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.375117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.375089 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:31.467801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.467756 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" event={"ID":"a00de023-dcb2-4a2a-baf4-396b1897857a","Type":"ContainerStarted","Data":"7d51e91cdacf8c9dfa2e1a3a59a36207b31b4cff52f0baa1037dc1166b251478"} Oct 31 00:31:31.469083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.469041 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerStarted","Data":"27cb9c05f1ad7ff548a73479989c98939da0bef97ca32ad854fd1dd3251d9047"} Oct 31 00:31:31.470323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.470302 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" event={"ID":"40cae269-904f-4533-baff-d7af79d3e111","Type":"ContainerStarted","Data":"ca33108a5b586cba7208e315d86432a99665c5f59253188ef07f928e708a8168"} Oct 31 00:31:31.674964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:31.674891 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:31:31.677846 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:31:31.677814 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7f3b9eb_9946_4e6f_af8b_847266f19efd.slice/crio-0e9a8266e2d1f84c75b28ae31c011a92c01d7018d4e2d0f28f0e247ee07ce23a WatchSource:0}: Error finding container 0e9a8266e2d1f84c75b28ae31c011a92c01d7018d4e2d0f28f0e247ee07ce23a: Status 404 returned error can't find the container with id 0e9a8266e2d1f84c75b28ae31c011a92c01d7018d4e2d0f28f0e247ee07ce23a Oct 31 00:31:32.194204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.194168 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:32.316609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.316572 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:32.476624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.476540 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"f7f3b9eb-9946-4e6f-af8b-847266f19efd","Type":"ContainerStarted","Data":"7d182ad17dad5e250161c7512488123b2f96853ca9c25e5aee2242274813bec4"} Oct 31 00:31:32.476624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.476584 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"f7f3b9eb-9946-4e6f-af8b-847266f19efd","Type":"ContainerStarted","Data":"0e9a8266e2d1f84c75b28ae31c011a92c01d7018d4e2d0f28f0e247ee07ce23a"} Oct 31 00:31:32.477026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.476659 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:32.494695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:32.494628 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=5.49461193 podStartE2EDuration="5.49461193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:32.494391412 +0000 UTC m=+1931.486051008" watchObservedRunningTime="2026-10-31 00:31:32.49461193 +0000 UTC m=+1931.486271513" Oct 31 00:31:33.139862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:33.139721 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:37.500161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.500124 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" event={"ID":"40cae269-904f-4533-baff-d7af79d3e111","Type":"ContainerStarted","Data":"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348"} Oct 31 00:31:37.500635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.500269 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:31:37.501998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.501967 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" event={"ID":"a00de023-dcb2-4a2a-baf4-396b1897857a","Type":"ContainerStarted","Data":"c6ddf871e2ee78f6e92f3f0cce58e77fac328dcad9ef7909e8f9932cf3d18ba8"} Oct 31 00:31:37.502135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.502112 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:37.521271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.520582 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podStartSLOduration=1.502237161 podStartE2EDuration="7.520569962s" totalImagesPullingTime="6.018332801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:37.519877863 +0000 UTC m=+1936.511537464" watchObservedRunningTime="2026-10-31 00:31:37.520569962 +0000 UTC m=+1936.512229542" Oct 31 00:31:37.539455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:37.539405 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" podStartSLOduration=2.496027846 podStartE2EDuration="8.539392318s" totalImagesPullingTime="6.043364472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:37.538775191 +0000 UTC m=+1936.530434776" watchObservedRunningTime="2026-10-31 00:31:37.539392318 +0000 UTC m=+1936.531051901" Oct 31 00:31:38.507400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:38.507358 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerStarted","Data":"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701"} Oct 31 00:31:41.522788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:41.522749 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerStarted","Data":"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d"} Oct 31 00:31:41.523425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:41.522911 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:41.524265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:41.524242 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.73:8083/ready\": dial tcp 10.133.0.73:8083: connect: connection refused" Oct 31 00:31:41.547342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:41.547280 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" podStartSLOduration=1.908167824 podStartE2EDuration="11.547264118s" totalImagesPullingTime="9.639096294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:41.54719252 +0000 UTC m=+1940.538852104" watchObservedRunningTime="2026-10-31 00:31:41.547264118 +0000 UTC m=+1940.538923703" Oct 31 00:31:42.526723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:42.526674 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:42.528491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:42.528468 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:42.566088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:42.566063 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:43.057482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:43.057450 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:43.482743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:43.482653 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:31:44.535494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:44.535457 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway" containerID="cri-o://87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701" gracePeriod=30 Oct 31 00:31:44.535959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:44.535493 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway-opa" containerID="cri-o://9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d" gracePeriod=30 Oct 31 00:31:45.541496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.541463 5577 generic.go:356] "Generic (PLEG): container finished" podID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerID="9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d" exitCode=0 Oct 31 00:31:45.541918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.541536 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerDied","Data":"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d"} Oct 31 00:31:45.780799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.780775 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:45.875392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875309 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875348 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\" (UniqueName: \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875391 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875426 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875447 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875471 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875498 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\") pod \"f89856e3-8a50-40b0-a224-e86e999bf6d6\" (UID: \"f89856e3-8a50-40b0-a224-e86e999bf6d6\") " Oct 31 00:31:45.875904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875832 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:45.875904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.875874 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:45.876092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.876065 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:45.877677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.877651 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:45.877810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.877724 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:45.878161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.878143 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "kube-api-access-x4r7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:45.881421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.881401 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant" pod "f89856e3-8a50-40b0-a224-e86e999bf6d6" (UID: "f89856e3-8a50-40b0-a224-e86e999bf6d6"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:45.976982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.976952 5577 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-serving-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.976982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.976980 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4r7z\" (UniqueName: \"kubernetes.io/projected/f89856e3-8a50-40b0-a224-e86e999bf6d6-kube-api-access-x4r7z\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.977146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.976992 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.977146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.977003 5577 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-cabundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.977146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.977012 5577 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tenant\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.977146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.977021 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f89856e3-8a50-40b0-a224-e86e999bf6d6-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:45.977146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:45.977029 5577 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f89856e3-8a50-40b0-a224-e86e999bf6d6-rbac\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:46.547151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.547118 5577 generic.go:356] "Generic (PLEG): container finished" podID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerID="87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701" exitCode=0 Oct 31 00:31:46.547539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.547168 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" Oct 31 00:31:46.547539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.547171 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerDied","Data":"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701"} Oct 31 00:31:46.547539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.547295 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs" event={"ID":"f89856e3-8a50-40b0-a224-e86e999bf6d6","Type":"ContainerDied","Data":"27cb9c05f1ad7ff548a73479989c98939da0bef97ca32ad854fd1dd3251d9047"} Oct 31 00:31:46.547539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.547327 5577 scope.go:122] "RemoveContainer" containerID="9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d" Oct 31 00:31:46.558495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.558437 5577 scope.go:122] "RemoveContainer" containerID="87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701" Oct 31 00:31:46.566821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.566801 5577 scope.go:122] "RemoveContainer" containerID="9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d" Oct 31 00:31:46.567063 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:31:46.567044 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d\": container with ID starting with 9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d not found: ID does not exist" containerID="9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d" Oct 31 00:31:46.567126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.567071 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d"} err="failed to get container status \"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d\": rpc error: code = NotFound desc = could not find container \"9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d\": container with ID starting with 9090b4073a6e5b6714a36b15ed36d77044c62de9d5e190563cd927715d4de97d not found: ID does not exist" Oct 31 00:31:46.567126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.567088 5577 scope.go:122] "RemoveContainer" containerID="87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701" Oct 31 00:31:46.567323 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:31:46.567304 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701\": container with ID starting with 87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701 not found: ID does not exist" containerID="87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701" Oct 31 00:31:46.567362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.567329 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701"} err="failed to get container status \"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701\": rpc error: code = NotFound desc = could not find container \"87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701\": container with ID starting with 87b94f6c287941591b74528bcfa02a5e3c13e5d11ad344d43184791efba3a701 not found: ID does not exist" Oct 31 00:31:46.571701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.571681 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:46.577230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:46.577211 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-849f997544-j6lbs"] Oct 31 00:31:47.129131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:47.129101 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" path="/var/lib/kubelet/pods/f89856e3-8a50-40b0-a224-e86e999bf6d6/volumes" Oct 31 00:31:52.590769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:52.590703 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:31:58.513036 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:58.513005 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:31:58.513447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:31:58.513285 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:31:59.292076 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:31:59.292037 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:32:08.512350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:08.512312 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:18.110142 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:18.110104 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:18.513639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:18.513592 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:26.289043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:26.289011 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:28.512598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:28.512516 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:29.292083 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:32:29.292049 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:32:38.512155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:38.512112 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:48.512088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:48.512053 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:32:50.003266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:50.003215 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:50.138318 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:50.138265 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:50.297401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:50.297319 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:50.420530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:50.420498 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.650950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.650907 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.655900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.655868 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:32:51.656141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.656116 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" containerName="perses" containerID="cri-o://7d182ad17dad5e250161c7512488123b2f96853ca9c25e5aee2242274813bec4" gracePeriod=30 Oct 31 00:32:51.790106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.790025 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.819348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.819305 5577 generic.go:356] "Generic (PLEG): container finished" podID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" containerID="7d182ad17dad5e250161c7512488123b2f96853ca9c25e5aee2242274813bec4" exitCode=0 Oct 31 00:32:51.819504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.819372 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"f7f3b9eb-9946-4e6f-af8b-847266f19efd","Type":"ContainerDied","Data":"7d182ad17dad5e250161c7512488123b2f96853ca9c25e5aee2242274813bec4"} Oct 31 00:32:51.834846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.834814 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:32:51.835027 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.834998 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" containerID="cri-o://c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348" gracePeriod=30 Oct 31 00:32:51.838103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.837553 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:32:51.838103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.837773 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" containerID="cri-o://c6ddf871e2ee78f6e92f3f0cce58e77fac328dcad9ef7909e8f9932cf3d18ba8" gracePeriod=30 Oct 31 00:32:51.922201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.922063 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:32:51.946468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:51.946439 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:32:52.005650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.005623 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\" (UniqueName: \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\") pod \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " Oct 31 00:32:52.005860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.005659 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\" (UniqueName: \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\") pod \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " Oct 31 00:32:52.005860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.005821 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " Oct 31 00:32:52.005992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.005872 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\" (UniqueName: \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\") pod \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\" (UID: \"f7f3b9eb-9946-4e6f-af8b-847266f19efd\") " Oct 31 00:32:52.006272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.006241 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config" pod "f7f3b9eb-9946-4e6f-af8b-847266f19efd" (UID: "f7f3b9eb-9946-4e6f-af8b-847266f19efd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:52.008210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.008178 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh" pod "f7f3b9eb-9946-4e6f-af8b-847266f19efd" (UID: "f7f3b9eb-9946-4e6f-af8b-847266f19efd"). InnerVolumeSpecName "kube-api-access-cbkvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:52.008636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.008618 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "f7f3b9eb-9946-4e6f-af8b-847266f19efd" (UID: "f7f3b9eb-9946-4e6f-af8b-847266f19efd"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:32:52.107345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.107309 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:32:52.107345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.107340 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7f3b9eb-9946-4e6f-af8b-847266f19efd-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.107345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.107351 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbkvh\" (UniqueName: \"kubernetes.io/projected/f7f3b9eb-9946-4e6f-af8b-847266f19efd-kube-api-access-cbkvh\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.120322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.120301 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:32:52.149436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.149408 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins" pod "f7f3b9eb-9946-4e6f-af8b-847266f19efd" (UID: "f7f3b9eb-9946-4e6f-af8b-847266f19efd"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:52.208219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.208147 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/f7f3b9eb-9946-4e6f-af8b-847266f19efd-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.208219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.208172 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:32:52.823828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.823791 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"f7f3b9eb-9946-4e6f-af8b-847266f19efd","Type":"ContainerDied","Data":"0e9a8266e2d1f84c75b28ae31c011a92c01d7018d4e2d0f28f0e247ee07ce23a"} Oct 31 00:32:52.823828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.823801 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:32:52.824334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.823833 5577 scope.go:122] "RemoveContainer" containerID="7d182ad17dad5e250161c7512488123b2f96853ca9c25e5aee2242274813bec4" Oct 31 00:32:52.846481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.846459 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:32:52.852600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:52.852579 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:32:53.129830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.129799 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" path="/var/lib/kubelet/pods/f7f3b9eb-9946-4e6f-af8b-847266f19efd/volumes" Oct 31 00:32:53.481858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.481835 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:32:53.499399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.499375 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:32:53.499537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.499520 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/seaweedfs" podUID="0bba5645-155b-4d63-957a-ba3489aefcd0" containerName="seaweedfs" containerID="cri-o://03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d" gracePeriod=30 Oct 31 00:32:53.620446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620404 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\") pod \"40cae269-904f-4533-baff-d7af79d3e111\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " Oct 31 00:32:53.620636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620457 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\" (UniqueName: \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\") pod \"40cae269-904f-4533-baff-d7af79d3e111\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " Oct 31 00:32:53.620636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620487 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\") pod \"40cae269-904f-4533-baff-d7af79d3e111\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " Oct 31 00:32:53.620636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620520 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\") pod \"40cae269-904f-4533-baff-d7af79d3e111\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " Oct 31 00:32:53.620636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620545 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\") pod \"40cae269-904f-4533-baff-d7af79d3e111\" (UID: \"40cae269-904f-4533-baff-d7af79d3e111\") " Oct 31 00:32:53.620893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620873 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle" pod "40cae269-904f-4533-baff-d7af79d3e111" (UID: "40cae269-904f-4533-baff-d7af79d3e111"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.620963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620938 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage" pod "40cae269-904f-4533-baff-d7af79d3e111" (UID: "40cae269-904f-4533-baff-d7af79d3e111"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:53.621026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.620968 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf" pod "40cae269-904f-4533-baff-d7af79d3e111" (UID: "40cae269-904f-4533-baff-d7af79d3e111"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:53.622840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.622816 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l" pod "40cae269-904f-4533-baff-d7af79d3e111" (UID: "40cae269-904f-4533-baff-d7af79d3e111"). InnerVolumeSpecName "kube-api-access-99v4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:53.622937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.622922 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls" pod "40cae269-904f-4533-baff-d7af79d3e111" (UID: "40cae269-904f-4533-baff-d7af79d3e111"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:53.722067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.721982 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.722067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.722019 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99v4l\" (UniqueName: \"kubernetes.io/projected/40cae269-904f-4533-baff-d7af79d3e111-kube-api-access-99v4l\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.722067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.722029 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/40cae269-904f-4533-baff-d7af79d3e111-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.722067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.722039 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/40cae269-904f-4533-baff-d7af79d3e111-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.722067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.722049 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/40cae269-904f-4533-baff-d7af79d3e111-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:53.829726 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.829674 5577 generic.go:356] "Generic (PLEG): container finished" podID="40cae269-904f-4533-baff-d7af79d3e111" containerID="c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348" exitCode=0 Oct 31 00:32:53.830099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.829747 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" Oct 31 00:32:53.830099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.829748 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" event={"ID":"40cae269-904f-4533-baff-d7af79d3e111","Type":"ContainerDied","Data":"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348"} Oct 31 00:32:53.830099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.829804 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws" event={"ID":"40cae269-904f-4533-baff-d7af79d3e111","Type":"ContainerDied","Data":"ca33108a5b586cba7208e315d86432a99665c5f59253188ef07f928e708a8168"} Oct 31 00:32:53.830099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.829828 5577 scope.go:122] "RemoveContainer" containerID="c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348" Oct 31 00:32:53.840935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.840911 5577 scope.go:122] "RemoveContainer" containerID="c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348" Oct 31 00:32:53.841335 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:32:53.841316 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348\": container with ID starting with c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348 not found: ID does not exist" containerID="c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348" Oct 31 00:32:53.841434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.841347 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348"} err="failed to get container status \"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348\": rpc error: code = NotFound desc = could not find container \"c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348\": container with ID starting with c595534123d1cfa7d257d3fb27e2d3d9b0feef37f3d2f2948b6fefe70b373348 not found: ID does not exist" Oct 31 00:32:53.851641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.851617 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:32:53.855922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:53.855897 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-7898846dc-64sws"] Oct 31 00:32:55.129608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:55.129566 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40cae269-904f-4533-baff-d7af79d3e111" path="/var/lib/kubelet/pods/40cae269-904f-4533-baff-d7af79d3e111/volumes" Oct 31 00:32:58.511751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:32:58.511694 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:59.291050 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:32:59.291007 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:33:08.511988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:08.511945 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:18.511624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.511588 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:33:18.512058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.511741 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:33:18.928657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.928635 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:33:18.931502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.931478 5577 generic.go:356] "Generic (PLEG): container finished" podID="0bba5645-155b-4d63-957a-ba3489aefcd0" containerID="03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d" exitCode=0 Oct 31 00:33:18.931621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.931511 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"0bba5645-155b-4d63-957a-ba3489aefcd0","Type":"ContainerDied","Data":"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d"} Oct 31 00:33:18.931621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.931516 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs" Oct 31 00:33:18.931621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.931536 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs" event={"ID":"0bba5645-155b-4d63-957a-ba3489aefcd0","Type":"ContainerDied","Data":"1150f04aa9db56229032fa35a7e20ea64ddddf4ad4555be3a2129ef62a77633d"} Oct 31 00:33:18.931621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.931552 5577 scope.go:122] "RemoveContainer" containerID="03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d" Oct 31 00:33:18.939761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.939739 5577 scope.go:122] "RemoveContainer" containerID="03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d" Oct 31 00:33:18.940054 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:33:18.940030 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d\": container with ID starting with 03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d not found: ID does not exist" containerID="03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d" Oct 31 00:33:18.940130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:18.940068 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d"} err="failed to get container status \"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d\": rpc error: code = NotFound desc = could not find container \"03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d\": container with ID starting with 03fcfe314acb6db0fd3f652278b53a602167cbfe3342b9bd340e4601fafbf58d not found: ID does not exist" Oct 31 00:33:19.015903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.015873 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\" (UniqueName: \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\") pod \"0bba5645-155b-4d63-957a-ba3489aefcd0\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " Oct 31 00:33:19.016086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.015960 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\" (UniqueName: \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\") pod \"0bba5645-155b-4d63-957a-ba3489aefcd0\" (UID: \"0bba5645-155b-4d63-957a-ba3489aefcd0\") " Oct 31 00:33:19.016879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.016852 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data" pod "0bba5645-155b-4d63-957a-ba3489aefcd0" (UID: "0bba5645-155b-4d63-957a-ba3489aefcd0"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:19.018140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.018115 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb" pod "0bba5645-155b-4d63-957a-ba3489aefcd0" (UID: "0bba5645-155b-4d63-957a-ba3489aefcd0"). InnerVolumeSpecName "kube-api-access-zhrfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:19.117012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.116951 5577 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0bba5645-155b-4d63-957a-ba3489aefcd0-data\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:19.117012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.116976 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhrfb\" (UniqueName: \"kubernetes.io/projected/0bba5645-155b-4d63-957a-ba3489aefcd0-kube-api-access-zhrfb\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:19.247624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.247587 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:33:19.252087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.252065 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs"] Oct 31 00:33:19.883604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:19.883569 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:21.129404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:21.129369 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bba5645-155b-4d63-957a-ba3489aefcd0" path="/var/lib/kubelet/pods/0bba5645-155b-4d63-957a-ba3489aefcd0/volumes" Oct 31 00:33:21.950963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:21.950929 5577 generic.go:356] "Generic (PLEG): container finished" podID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerID="c6ddf871e2ee78f6e92f3f0cce58e77fac328dcad9ef7909e8f9932cf3d18ba8" exitCode=137 Oct 31 00:33:21.951111 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:21.951001 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" event={"ID":"a00de023-dcb2-4a2a-baf4-396b1897857a","Type":"ContainerDied","Data":"c6ddf871e2ee78f6e92f3f0cce58e77fac328dcad9ef7909e8f9932cf3d18ba8"} Oct 31 00:33:21.981523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:21.981503 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:33:22.114207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114130 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:33:22.114421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114407 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0bba5645-155b-4d63-957a-ba3489aefcd0" containerName="seaweedfs" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114424 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bba5645-155b-4d63-957a-ba3489aefcd0" containerName="seaweedfs" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114433 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114438 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114457 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114461 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway" Oct 31 00:33:22.114472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114471 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway-opa" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114477 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway-opa" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114483 5577 cpu_manager.go:414] "containerMap: removing container" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114487 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114496 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" containerName="perses" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114501 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" containerName="perses" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114543 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114551 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f89856e3-8a50-40b0-a224-e86e999bf6d6" containerName="tempo-gateway-opa" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114557 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="40cae269-904f-4533-baff-d7af79d3e111" containerName="tempo" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114564 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f3b9eb-9946-4e6f-af8b-847266f19efd" containerName="perses" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114570 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bba5645-155b-4d63-957a-ba3489aefcd0" containerName="seaweedfs" Oct 31 00:33:22.114660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.114575 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" containerName="tempo" Oct 31 00:33:22.117287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.117273 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.121166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.121142 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:33:22.125979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.125955 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:33:22.140216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140186 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\") pod \"a00de023-dcb2-4a2a-baf4-396b1897857a\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " Oct 31 00:33:22.140603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140266 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\") pod \"a00de023-dcb2-4a2a-baf4-396b1897857a\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " Oct 31 00:33:22.140603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140330 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\") pod \"a00de023-dcb2-4a2a-baf4-396b1897857a\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " Oct 31 00:33:22.140603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140358 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\" (UniqueName: \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\") pod \"a00de023-dcb2-4a2a-baf4-396b1897857a\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " Oct 31 00:33:22.140603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140429 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\") pod \"a00de023-dcb2-4a2a-baf4-396b1897857a\" (UID: \"a00de023-dcb2-4a2a-baf4-396b1897857a\") " Oct 31 00:33:22.140603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140517 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage" pod "a00de023-dcb2-4a2a-baf4-396b1897857a" (UID: "a00de023-dcb2-4a2a-baf4-396b1897857a"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:33:22.140901 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140644 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.141119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140961 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf" pod "a00de023-dcb2-4a2a-baf4-396b1897857a" (UID: "a00de023-dcb2-4a2a-baf4-396b1897857a"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:22.141119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.140975 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle" pod "a00de023-dcb2-4a2a-baf4-396b1897857a" (UID: "a00de023-dcb2-4a2a-baf4-396b1897857a"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:22.142556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.142531 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls" pod "a00de023-dcb2-4a2a-baf4-396b1897857a" (UID: "a00de023-dcb2-4a2a-baf4-396b1897857a"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:22.142877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.142857 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2" pod "a00de023-dcb2-4a2a-baf4-396b1897857a" (UID: "a00de023-dcb2-4a2a-baf4-396b1897857a"). InnerVolumeSpecName "kube-api-access-bvqb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:22.226178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.226136 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:22.241146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241116 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.241251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241179 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gpm\" (UniqueName: \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.241294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241258 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.241294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241275 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.241294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241287 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvqb2\" (UniqueName: \"kubernetes.io/projected/a00de023-dcb2-4a2a-baf4-396b1897857a-kube-api-access-bvqb2\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.241392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.241298 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a00de023-dcb2-4a2a-baf4-396b1897857a-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:22.342187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.342163 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.342319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.342208 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gpm\" (UniqueName: \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.342530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.342513 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.355392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.355366 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gpm\" (UniqueName: \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\") pod \"fake-gcs-server\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.428518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.428454 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:22.698406 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.698333 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:33:22.701427 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:22.701397 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod75367784_658e_432c_9095_5ccb67fe1207.slice/crio-cadde8b102f1fe140f3cbb4011a1683442cdee4f6aa27e85e1ed816514a2d520 WatchSource:0}: Error finding container cadde8b102f1fe140f3cbb4011a1683442cdee4f6aa27e85e1ed816514a2d520: Status 404 returned error can't find the container with id cadde8b102f1fe140f3cbb4011a1683442cdee4f6aa27e85e1ed816514a2d520 Oct 31 00:33:22.955971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.955890 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"75367784-658e-432c-9095-5ccb67fe1207","Type":"ContainerStarted","Data":"cadde8b102f1fe140f3cbb4011a1683442cdee4f6aa27e85e1ed816514a2d520"} Oct 31 00:33:22.957098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.957076 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" event={"ID":"a00de023-dcb2-4a2a-baf4-396b1897857a","Type":"ContainerDied","Data":"7d51e91cdacf8c9dfa2e1a3a59a36207b31b4cff52f0baa1037dc1166b251478"} Oct 31 00:33:22.957198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.957122 5577 scope.go:122] "RemoveContainer" containerID="c6ddf871e2ee78f6e92f3f0cce58e77fac328dcad9ef7909e8f9932cf3d18ba8" Oct 31 00:33:22.957198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.957087 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb" Oct 31 00:33:22.978347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.978326 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:33:22.982843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:22.982823 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-5b7b64f477-5r6wb"] Oct 31 00:33:23.129462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:23.129425 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00de023-dcb2-4a2a-baf4-396b1897857a" path="/var/lib/kubelet/pods/a00de023-dcb2-4a2a-baf4-396b1897857a/volumes" Oct 31 00:33:23.964102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:23.964061 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"75367784-658e-432c-9095-5ccb67fe1207","Type":"ContainerStarted","Data":"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd"} Oct 31 00:33:23.964508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:23.964109 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:23.983772 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:23.983725 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/fake-gcs-server" podStartSLOduration=0.85843649 podStartE2EDuration="1.983689097s" totalImagesPullingTime="1.125252607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:23.983352928 +0000 UTC m=+2042.975012537" watchObservedRunningTime="2026-10-31 00:33:23.983689097 +0000 UTC m=+2042.975348678" Oct 31 00:33:26.340495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:26.340464 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:29.291187 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:33:29.291154 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:33:29.976312 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:29.976280 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:33:35.191308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.191275 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.336302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.336261 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.479612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.479544 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:33:35.482890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.482869 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.487961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.487936 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.492966 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.492944 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:33:35.556745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.556697 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:33:35.617978 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.617928 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.651559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.651535 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxkmw\" (UniqueName: \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.651668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.651591 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.651729 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.651683 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.742065 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.741966 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:35.752266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.752227 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.752390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.752282 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxkmw\" (UniqueName: \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.752390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.752347 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.752576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.752559 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.753111 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.753089 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:35.765678 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:35.765655 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxkmw\" (UniqueName: \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:38.025320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.025289 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:33:38.029629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.029609 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.036515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.036493 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 31 00:33:38.039061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.039043 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 31 00:33:38.039892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.039870 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-4n979\"" Oct 31 00:33:38.041692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.039976 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 31 00:33:38.041692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.040899 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:33:38.060356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.060325 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:38.063901 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.063880 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.077183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.077158 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:38.109633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.109582 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.139433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.139407 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 31 00:33:38.139592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.139536 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 31 00:33:38.169321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169285 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.169462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169341 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwj6p\" (UniqueName: \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.169462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169414 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.169462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169456 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.169585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169476 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.169585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169565 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjzl\" (UniqueName: \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.169649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169627 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.169731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169687 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.169777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169740 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.169866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.169850 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.243760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.243723 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.271216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271142 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.271216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271189 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.271421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271305 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwj6p\" (UniqueName: \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.271421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271345 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.271421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271382 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.271421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271398 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.271631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271429 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjzl\" (UniqueName: \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.271631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271446 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.271631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271473 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.271631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271504 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.271872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.271833 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.272092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.272064 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.272636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.272612 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.272770 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.272612 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.272770 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.272726 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.274042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.274018 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.274168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.274149 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.274232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.274168 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.284374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.284351 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwj6p\" (UniqueName: \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:38.284639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.284616 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjzl\" (UniqueName: \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\") pod \"tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.346979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.346955 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:38.377411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.377382 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:33:38.539749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.539653 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:38.547268 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:33:38.547235 5577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[data], unattached volumes=[data], failed to process volumes=[]: context canceled" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" Oct 31 00:33:38.587575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.587550 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:33:38.592003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.591980 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.602292 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.601786 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:33:38.605482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.605461 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:33:38.605587 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.605567 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.618659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.618606 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:33:38.622907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.622888 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:33:38.623022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.622991 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.626568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.626551 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:33:38.630216 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:38.630190 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd775bbe0_668b_4a27_ae5c_18a28bf85470.slice/crio-d56e2f5a17543f0b86ee8b7d64e2ac1c1b7bac7b43231ca4dac51bbf93661e56 WatchSource:0}: Error finding container d56e2f5a17543f0b86ee8b7d64e2ac1c1b7bac7b43231ca4dac51bbf93661e56: Status 404 returned error can't find the container with id d56e2f5a17543f0b86ee8b7d64e2ac1c1b7bac7b43231ca4dac51bbf93661e56 Oct 31 00:33:38.635857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.635235 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:33:38.663901 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.663876 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 31 00:33:38.664026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.663999 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 31 00:33:38.664223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.664093 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 31 00:33:38.668869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.668847 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:33:38.674042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.674023 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.681357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.681339 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:33:38.681535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.681520 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 31 00:33:38.687432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.687410 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:33:38.764472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.764441 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 31 00:33:38.764638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.764480 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 31 00:33:38.764638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.764536 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 31 00:33:38.764638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.764611 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-rnszp\"" Oct 31 00:33:38.775051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775025 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775059 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgbfp\" (UniqueName: \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775087 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.775272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775250 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775302 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775338 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.775394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775365 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775400 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775441 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775467 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfp85\" (UniqueName: \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775499 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xldcl\" (UniqueName: \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.775588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775554 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775576 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775605 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775650 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.775745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775668 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.775745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775689 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775739 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775770 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775803 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775824 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.775961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775866 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc68q\" (UniqueName: \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775887 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.775961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775905 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.775961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.775937 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877053 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.877131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877085 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877102 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877135 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877162 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgbfp\" (UniqueName: \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877191 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877247 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877282 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877305 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877333 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877359 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.877411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877388 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877414 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfp85\" (UniqueName: \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877439 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xldcl\" (UniqueName: \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877465 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877491 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877533 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877538 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877569 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877595 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877621 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.877891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877684 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.878450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.877915 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.878450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878007 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.878450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878031 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.878450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878058 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc68q\" (UniqueName: \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.878450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878164 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.878816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878789 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.878989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.878818 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.879084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.879007 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.879420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.879394 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.879759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.879732 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.880138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.880111 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.880239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.880066 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.880370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.880351 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.880540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.880516 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.881252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.881222 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.881379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.881352 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.881863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.881838 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.882189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.882168 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.882580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.882556 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.883158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.883137 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.883395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.883375 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.883697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.883681 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.883821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.883801 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.883999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.883977 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.892930 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.892903 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc68q\" (UniqueName: \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\") pod \"tempo-data-science-tempostack-gateway-5b78f57785-7kbwt\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:38.893043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.893025 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xldcl\" (UniqueName: \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\") pod \"tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.893136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.893119 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgbfp\" (UniqueName: \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\") pod \"tempo-data-science-tempostack-compactor-574d99548c-h88md\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:38.893187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.893135 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfp85\" (UniqueName: \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\") pod \"tempo-data-science-tempostack-querier-84dcb46bf9-t89sd\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.907515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.907500 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:38.920010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.919985 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:38.939881 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:38.939849 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:39.002180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.001887 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:39.033407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.033227 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:39.033407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.033267 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" event={"ID":"d775bbe0-668b-4a27-ae5c-18a28bf85470","Type":"ContainerStarted","Data":"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb"} Oct 31 00:33:39.033407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.033327 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" event={"ID":"d775bbe0-668b-4a27-ae5c-18a28bf85470","Type":"ContainerStarted","Data":"d56e2f5a17543f0b86ee8b7d64e2ac1c1b7bac7b43231ca4dac51bbf93661e56"} Oct 31 00:33:39.033407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.033360 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:33:39.043825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.043800 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:39.077392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.077169 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" podStartSLOduration=1.0771450439999999 podStartE2EDuration="1.077145044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:39.053665665 +0000 UTC m=+2058.045325250" watchObservedRunningTime="2026-10-31 00:33:39.077145044 +0000 UTC m=+2058.068804632" Oct 31 00:33:39.180883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.180820 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\" (UniqueName: \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\") pod \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " Oct 31 00:33:39.180883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.180874 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\") pod \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " Oct 31 00:33:39.181097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.180917 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\") pod \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " Oct 31 00:33:39.181097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.180956 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\") pod \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " Oct 31 00:33:39.181097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.181076 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\") pod \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\" (UID: \"318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5\") " Oct 31 00:33:39.181330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.181299 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle" pod "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" (UID: "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:39.181401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.181349 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf" pod "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" (UID: "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:33:39.182843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.181469 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.182843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.181492 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.183435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.183407 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p" pod "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" (UID: "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5"). InnerVolumeSpecName "kube-api-access-fwj6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:33:39.183615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.183591 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key" pod "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" (UID: "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:39.183673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.183656 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls" pod "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" (UID: "318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:33:39.183740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.183725 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:33:39.186563 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:39.186539 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ecd7df_e24c_41cb_bebf_3ffaf8063024.slice/crio-edf9bfedfe1f46bde567ae2b28d325d47b24e46fdd91fa4b80e3699b6bec558e WatchSource:0}: Error finding container edf9bfedfe1f46bde567ae2b28d325d47b24e46fdd91fa4b80e3699b6bec558e: Status 404 returned error can't find the container with id edf9bfedfe1f46bde567ae2b28d325d47b24e46fdd91fa4b80e3699b6bec558e Oct 31 00:33:39.216244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.215927 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:33:39.218042 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:39.218019 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4187c9_ce3c_4035_83ed_6cafdffb3a69.slice/crio-4bd1b5eb3b3e4136117b84558ff5fd775194270b5b61bb1dfbfe2c42540b6f02 WatchSource:0}: Error finding container 4bd1b5eb3b3e4136117b84558ff5fd775194270b5b61bb1dfbfe2c42540b6f02: Status 404 returned error can't find the container with id 4bd1b5eb3b3e4136117b84558ff5fd775194270b5b61bb1dfbfe2c42540b6f02 Oct 31 00:33:39.233342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.233270 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:33:39.235652 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:39.235620 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6954d991_78f1_4807_8ea7_db4ef795d2a7.slice/crio-3087babf9ffe3e63373c4b6d4868db9ac36435045366b8c30b5a34f7ab8f6cf9 WatchSource:0}: Error finding container 3087babf9ffe3e63373c4b6d4868db9ac36435045366b8c30b5a34f7ab8f6cf9: Status 404 returned error can't find the container with id 3087babf9ffe3e63373c4b6d4868db9ac36435045366b8c30b5a34f7ab8f6cf9 Oct 31 00:33:39.282845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.282815 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.282845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.282845 5577 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-storage-gcs-key\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.282971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.282860 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwj6p\" (UniqueName: \"kubernetes.io/projected/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5-kube-api-access-fwj6p\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:33:39.291680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:39.291647 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:33:39.367012 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:39.366984 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda214686e_d4fc_467e_8ce7_c6858bc4c898.slice/crio-e889ddea1344b6c92a7b4419c085ba4229418a940fa92c0288fad4eb8b6858e3 WatchSource:0}: Error finding container e889ddea1344b6c92a7b4419c085ba4229418a940fa92c0288fad4eb8b6858e3: Status 404 returned error can't find the container with id e889ddea1344b6c92a7b4419c085ba4229418a940fa92c0288fad4eb8b6858e3 Oct 31 00:33:40.039162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.039073 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerStarted","Data":"c0ba7c42bb07d557e46cfaf0b47c1f2f1cfcaadfd38bad071d0c4001ad41e258"} Oct 31 00:33:40.039162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.039151 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:40.039631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.039184 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerStarted","Data":"4b7830bb51e6e9f6e7952f59f80273ae9980ffe9e15dcfbc50cef27d89016f4e"} Oct 31 00:33:40.039631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.039213 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:40.039631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.039235 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerStarted","Data":"e889ddea1344b6c92a7b4419c085ba4229418a940fa92c0288fad4eb8b6858e3"} Oct 31 00:33:40.040662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.040630 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.133.0.81:8081/ready\": dial tcp 10.133.0.81:8081: connect: connection refused" Oct 31 00:33:40.040831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.040635 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.81:8083/ready\": dial tcp 10.133.0.81:8083: connect: connection refused" Oct 31 00:33:40.040936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.040917 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" event={"ID":"82ecd7df-e24c-41cb-bebf-3ffaf8063024","Type":"ContainerStarted","Data":"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2"} Oct 31 00:33:40.041010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.040971 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:33:40.041010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.040997 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" event={"ID":"82ecd7df-e24c-41cb-bebf-3ffaf8063024","Type":"ContainerStarted","Data":"edf9bfedfe1f46bde567ae2b28d325d47b24e46fdd91fa4b80e3699b6bec558e"} Oct 31 00:33:40.042627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.042602 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" event={"ID":"6954d991-78f1-4807-8ea7-db4ef795d2a7","Type":"ContainerStarted","Data":"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5"} Oct 31 00:33:40.042797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.042654 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:33:40.042797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.042677 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" event={"ID":"6954d991-78f1-4807-8ea7-db4ef795d2a7","Type":"ContainerStarted","Data":"3087babf9ffe3e63373c4b6d4868db9ac36435045366b8c30b5a34f7ab8f6cf9"} Oct 31 00:33:40.044227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.044197 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" event={"ID":"be4187c9-ce3c-4035-83ed-6cafdffb3a69","Type":"ContainerStarted","Data":"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2"} Oct 31 00:33:40.044312 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.044227 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.044312 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.044237 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" event={"ID":"be4187c9-ce3c-4035-83ed-6cafdffb3a69","Type":"ContainerStarted","Data":"4bd1b5eb3b3e4136117b84558ff5fd775194270b5b61bb1dfbfe2c42540b6f02"} Oct 31 00:33:40.044532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.044516 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:33:40.063341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.063298 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podStartSLOduration=2.063287035 podStartE2EDuration="2.063287035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:40.06246465 +0000 UTC m=+2059.054124234" watchObservedRunningTime="2026-10-31 00:33:40.063287035 +0000 UTC m=+2059.054946619" Oct 31 00:33:40.095776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.095746 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:40.098829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.098798 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:40.117972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.117927 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" podStartSLOduration=2.117915638 podStartE2EDuration="2.117915638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:40.117157984 +0000 UTC m=+2059.108817567" watchObservedRunningTime="2026-10-31 00:33:40.117915638 +0000 UTC m=+2059.109575221" Oct 31 00:33:40.129302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.129277 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:40.132607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.132591 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.137818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.137793 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ingester-mtls\"" Oct 31 00:33:40.144080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.144031 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podStartSLOduration=2.143981741 podStartE2EDuration="2.143981741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:40.137891539 +0000 UTC m=+2059.129551138" watchObservedRunningTime="2026-10-31 00:33:40.143981741 +0000 UTC m=+2059.135641326" Oct 31 00:33:40.144563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.144542 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:40.163343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.163295 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" podStartSLOduration=2.16328413 podStartE2EDuration="2.16328413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:33:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:40.162655935 +0000 UTC m=+2059.154315520" watchObservedRunningTime="2026-10-31 00:33:40.16328413 +0000 UTC m=+2059.154943705" Oct 31 00:33:40.292688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.292621 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qvhz\" (UniqueName: \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.292688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.292653 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.292688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.292682 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.292914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.292823 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.292914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.292850 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.393929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.393894 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qvhz\" (UniqueName: \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.393929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.393930 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.394118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.393957 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.394118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.394036 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.394118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.394071 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.394690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.394663 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.395011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.394842 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.396359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.396334 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.396644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.396623 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:40.406917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:40.406894 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qvhz\" (UniqueName: \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:41.049062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:41.049020 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.81:8083/ready\": dial tcp 10.133.0.81:8083: connect: connection refused" Oct 31 00:33:41.130879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:41.130849 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5" path="/var/lib/kubelet/pods/318d16f0-dcc3-4f2e-a9fe-a7111ddf84f5/volumes" Oct 31 00:33:41.766384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:41.766348 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:42.053970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:42.053895 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:33:43.725209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.725178 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.725621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.725238 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.734345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.734315 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.734345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.734321 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.825968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.825938 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.826143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.825991 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.826143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.826102 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.826143 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.826106 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.830001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.829979 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.830088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.830036 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") DevicePath \"csi-f05e1daa8a0e8f58357f7c0c4175de250b15de8ccf16d9c7ba80e381afe95735\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.923543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.923506 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:43.924262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.924233 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/1018b4fc58a16f6eda6b1e7c04ae4351db82acf0495116e38742f3ca07150739/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.939635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.939609 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"tempo-data-science-tempostack-ingester-0\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:43.942952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:43.942920 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:44.045209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:44.045179 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:44.195560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:44.195523 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:44.322017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:44.321975 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:33:44.324103 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:44.324075 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2663b543_da9f_417d_b839_08f693ee8298.slice/crio-e6414d42de440b02abb2d9bd89b1f9e47bc7efc257b8b265bb37df644bac589b WatchSource:0}: Error finding container e6414d42de440b02abb2d9bd89b1f9e47bc7efc257b8b265bb37df644bac589b: Status 404 returned error can't find the container with id e6414d42de440b02abb2d9bd89b1f9e47bc7efc257b8b265bb37df644bac589b Oct 31 00:33:44.474609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:44.474574 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:33:44.477321 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:33:44.477294 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6f20fb0_fda1_4580_b4da_341d02b65a87.slice/crio-35c619537d2ffb34b4a2d0887bb10e5b230b8c514a3a16f704c0b96a826a5424 WatchSource:0}: Error finding container 35c619537d2ffb34b4a2d0887bb10e5b230b8c514a3a16f704c0b96a826a5424: Status 404 returned error can't find the container with id 35c619537d2ffb34b4a2d0887bb10e5b230b8c514a3a16f704c0b96a826a5424 Oct 31 00:33:45.069780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.068733 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b6f20fb0-fda1-4580-b4da-341d02b65a87","Type":"ContainerStarted","Data":"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc"} Oct 31 00:33:45.069780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.068786 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b6f20fb0-fda1-4580-b4da-341d02b65a87","Type":"ContainerStarted","Data":"35c619537d2ffb34b4a2d0887bb10e5b230b8c514a3a16f704c0b96a826a5424"} Oct 31 00:33:45.069780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.068885 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:45.070658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.070634 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2663b543-da9f-417d-b839-08f693ee8298","Type":"ContainerStarted","Data":"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d"} Oct 31 00:33:45.070832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.070675 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:33:45.070832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.070691 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2663b543-da9f-417d-b839-08f693ee8298","Type":"ContainerStarted","Data":"e6414d42de440b02abb2d9bd89b1f9e47bc7efc257b8b265bb37df644bac589b"} Oct 31 00:33:45.089758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.088786 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=10.088772318 podStartE2EDuration="10.088772318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:33:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:45.088521306 +0000 UTC m=+2064.080180890" watchObservedRunningTime="2026-10-31 00:33:45.088772318 +0000 UTC m=+2064.080431900" Oct 31 00:33:45.115099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:45.115037 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podStartSLOduration=5.115019842 podStartE2EDuration="5.115019842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:33:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:33:45.113932863 +0000 UTC m=+2064.105592463" watchObservedRunningTime="2026-10-31 00:33:45.115019842 +0000 UTC m=+2064.106679426" Oct 31 00:33:56.076356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:33:56.076326 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:33:59.291821 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:33:59.291735 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:34:00.050259 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:00.050232 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:34:00.969419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:00.969383 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:34:00.969929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:00.969632 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" podUID="d775bbe0-668b-4a27-ae5c-18a28bf85470" containerName="tempo" containerID="cri-o://682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb" gracePeriod=30 Oct 31 00:34:01.055804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:01.055768 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:34:01.056002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:01.055935 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:01.166257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:01.166228 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:34:02.113093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.113068 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:34:02.149410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.149312 5577 generic.go:356] "Generic (PLEG): container finished" podID="d775bbe0-668b-4a27-ae5c-18a28bf85470" containerID="682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb" exitCode=0 Oct 31 00:34:02.149410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.149354 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" event={"ID":"d775bbe0-668b-4a27-ae5c-18a28bf85470","Type":"ContainerDied","Data":"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb"} Oct 31 00:34:02.149610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.149430 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" event={"ID":"d775bbe0-668b-4a27-ae5c-18a28bf85470","Type":"ContainerDied","Data":"d56e2f5a17543f0b86ee8b7d64e2ac1c1b7bac7b43231ca4dac51bbf93661e56"} Oct 31 00:34:02.149610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.149370 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc" Oct 31 00:34:02.149610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.149465 5577 scope.go:122] "RemoveContainer" containerID="682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb" Oct 31 00:34:02.159693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.159674 5577 scope.go:122] "RemoveContainer" containerID="682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb" Oct 31 00:34:02.160032 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:34:02.160013 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb\": container with ID starting with 682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb not found: ID does not exist" containerID="682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb" Oct 31 00:34:02.160087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.160041 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb"} err="failed to get container status \"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb\": rpc error: code = NotFound desc = could not find container \"682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb\": container with ID starting with 682755db5500c526aa20baaf474912ef9b393d7091b287a8a6bf6d276e1d8ffb not found: ID does not exist" Oct 31 00:34:02.183969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.183944 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\") pod \"d775bbe0-668b-4a27-ae5c-18a28bf85470\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " Oct 31 00:34:02.184124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184020 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\") pod \"d775bbe0-668b-4a27-ae5c-18a28bf85470\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " Oct 31 00:34:02.184124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184052 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\") pod \"d775bbe0-668b-4a27-ae5c-18a28bf85470\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " Oct 31 00:34:02.184124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184087 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\") pod \"d775bbe0-668b-4a27-ae5c-18a28bf85470\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " Oct 31 00:34:02.184124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184115 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\" (UniqueName: \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\") pod \"d775bbe0-668b-4a27-ae5c-18a28bf85470\" (UID: \"d775bbe0-668b-4a27-ae5c-18a28bf85470\") " Oct 31 00:34:02.184433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184392 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage" pod "d775bbe0-668b-4a27-ae5c-18a28bf85470" (UID: "d775bbe0-668b-4a27-ae5c-18a28bf85470"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:34:02.184549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184451 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf" pod "d775bbe0-668b-4a27-ae5c-18a28bf85470" (UID: "d775bbe0-668b-4a27-ae5c-18a28bf85470"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:02.184549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.184476 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle" pod "d775bbe0-668b-4a27-ae5c-18a28bf85470" (UID: "d775bbe0-668b-4a27-ae5c-18a28bf85470"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:34:02.186358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.186337 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl" pod "d775bbe0-668b-4a27-ae5c-18a28bf85470" (UID: "d775bbe0-668b-4a27-ae5c-18a28bf85470"). InnerVolumeSpecName "kube-api-access-vrjzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:34:02.186436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.186416 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls" pod "d775bbe0-668b-4a27-ae5c-18a28bf85470" (UID: "d775bbe0-668b-4a27-ae5c-18a28bf85470"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:34:02.284880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.284842 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:02.284880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.284874 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:02.284880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.284885 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:02.285127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.284894 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjzl\" (UniqueName: \"kubernetes.io/projected/d775bbe0-668b-4a27-ae5c-18a28bf85470-kube-api-access-vrjzl\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:02.285127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.284903 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/d775bbe0-668b-4a27-ae5c-18a28bf85470-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:34:02.472019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.471981 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:34:02.476401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:02.476374 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-75cfcbc9b6-wvlxc"] Oct 31 00:34:03.129458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:03.129411 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d775bbe0-668b-4a27-ae5c-18a28bf85470" path="/var/lib/kubelet/pods/d775bbe0-668b-4a27-ae5c-18a28bf85470/volumes" Oct 31 00:34:06.080839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:06.080805 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:09.436034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:09.435982 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:11.053972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:11.053921 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:16.078899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:16.078859 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:21.056185 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:21.056138 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:21.337603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:21.337528 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:34:21.342665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:21.342641 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:34:26.078163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:26.078132 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:34:26.395958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:26.395847 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:29.291589 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:34:29.291550 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:34:31.054167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:31.054124 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:41.054478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:41.054443 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:34:46.142360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:46.142313 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:51.053971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:51.053934 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:34:52.290509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:52.290475 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:52.413277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:52.413247 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:52.559877 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:52.559798 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:52.681774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:34:52.681701 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:34:59.291836 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:34:59.291803 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:35:24.102239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.102202 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:24.102850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.102465 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="b6f20fb0-fda1-4580-b4da-341d02b65a87" containerName="perses" containerID="cri-o://50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc" gracePeriod=30 Oct 31 00:35:24.132158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.132128 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:24.332791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.332751 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:35:24.333066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.333039 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" containerID="cri-o://f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d" gracePeriod=30 Oct 31 00:35:24.335671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.335647 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:35:24.335889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.335867 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" containerID="cri-o://44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2" gracePeriod=30 Oct 31 00:35:24.340899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.340880 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:35:24.343066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.343013 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" containerID="cri-o://b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5" gracePeriod=30 Oct 31 00:35:24.349945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.349919 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:35:24.350210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.350135 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" containerID="cri-o://1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2" gracePeriod=30 Oct 31 00:35:24.351139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.350974 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:35:24.351331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.351209 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway" containerID="cri-o://4b7830bb51e6e9f6e7952f59f80273ae9980ffe9e15dcfbc50cef27d89016f4e" gracePeriod=30 Oct 31 00:35:24.351331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.351251 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" containerID="cri-o://c0ba7c42bb07d557e46cfaf0b47c1f2f1cfcaadfd38bad071d0c4001ad41e258" gracePeriod=30 Oct 31 00:35:24.478871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.478844 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:24.505272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.505229 5577 generic.go:356] "Generic (PLEG): container finished" podID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerID="c0ba7c42bb07d557e46cfaf0b47c1f2f1cfcaadfd38bad071d0c4001ad41e258" exitCode=0 Oct 31 00:35:24.505444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.505269 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerDied","Data":"c0ba7c42bb07d557e46cfaf0b47c1f2f1cfcaadfd38bad071d0c4001ad41e258"} Oct 31 00:35:24.506792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.506768 5577 generic.go:356] "Generic (PLEG): container finished" podID="b6f20fb0-fda1-4580-b4da-341d02b65a87" containerID="50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc" exitCode=0 Oct 31 00:35:24.506929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.506811 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b6f20fb0-fda1-4580-b4da-341d02b65a87","Type":"ContainerDied","Data":"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc"} Oct 31 00:35:24.506929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.506824 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:24.506929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.506834 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"b6f20fb0-fda1-4580-b4da-341d02b65a87","Type":"ContainerDied","Data":"35c619537d2ffb34b4a2d0887bb10e5b230b8c514a3a16f704c0b96a826a5424"} Oct 31 00:35:24.506929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.506855 5577 scope.go:122] "RemoveContainer" containerID="50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc" Oct 31 00:35:24.517247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.517219 5577 scope.go:122] "RemoveContainer" containerID="50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc" Oct 31 00:35:24.517577 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:24.517551 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc\": container with ID starting with 50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc not found: ID does not exist" containerID="50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc" Oct 31 00:35:24.517665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.517589 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc"} err="failed to get container status \"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc\": rpc error: code = NotFound desc = could not find container \"50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc\": container with ID starting with 50c935d514e6aad03d1128221c437adb9a1be0a2009aa217b35e992081bb46dc not found: ID does not exist" Oct 31 00:35:24.598236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.598209 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"b6f20fb0-fda1-4580-b4da-341d02b65a87\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " Oct 31 00:35:24.598448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.598250 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\" (UniqueName: \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\") pod \"b6f20fb0-fda1-4580-b4da-341d02b65a87\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " Oct 31 00:35:24.598448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.598284 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\" (UniqueName: \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\") pod \"b6f20fb0-fda1-4580-b4da-341d02b65a87\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " Oct 31 00:35:24.598448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.598348 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\" (UniqueName: \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\") pod \"b6f20fb0-fda1-4580-b4da-341d02b65a87\" (UID: \"b6f20fb0-fda1-4580-b4da-341d02b65a87\") " Oct 31 00:35:24.598792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.598755 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config" pod "b6f20fb0-fda1-4580-b4da-341d02b65a87" (UID: "b6f20fb0-fda1-4580-b4da-341d02b65a87"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:24.600910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.600878 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw" pod "b6f20fb0-fda1-4580-b4da-341d02b65a87" (UID: "b6f20fb0-fda1-4580-b4da-341d02b65a87"). InnerVolumeSpecName "kube-api-access-rxkmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:24.601266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.601247 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "b6f20fb0-fda1-4580-b4da-341d02b65a87" (UID: "b6f20fb0-fda1-4580-b4da-341d02b65a87"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:35:24.699749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.699661 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:35:24.699749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.699687 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b6f20fb0-fda1-4580-b4da-341d02b65a87-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.699749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.699696 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxkmw\" (UniqueName: \"kubernetes.io/projected/b6f20fb0-fda1-4580-b4da-341d02b65a87-kube-api-access-rxkmw\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.712816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.712793 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:35:24.747440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.747410 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins" pod "b6f20fb0-fda1-4580-b4da-341d02b65a87" (UID: "b6f20fb0-fda1-4580-b4da-341d02b65a87"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:24.800382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.800345 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:35:24.800382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.800380 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/b6f20fb0-fda1-4580-b4da-341d02b65a87-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:24.828027 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.827993 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:24.836066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:24.836041 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:25.084952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.084930 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:35:25.130617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.130588 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f20fb0-fda1-4580-b4da-341d02b65a87" path="/var/lib/kubelet/pods/b6f20fb0-fda1-4580-b4da-341d02b65a87/volumes" Oct 31 00:35:25.203019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.202984 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203071 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203095 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\" (UniqueName: \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203114 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203149 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203183 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\") pod \"6954d991-78f1-4807-8ea7-db4ef795d2a7\" (UID: \"6954d991-78f1-4807-8ea7-db4ef795d2a7\") " Oct 31 00:35:25.203568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203540 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:25.203652 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203607 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.203652 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.203610 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.205419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.205398 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "kube-api-access-kgbfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:25.205497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.205414 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.205497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.205449 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key" pod "6954d991-78f1-4807-8ea7-db4ef795d2a7" (UID: "6954d991-78f1-4807-8ea7-db4ef795d2a7"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.304224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304189 5577 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-storage-gcs-key\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.304224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304218 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.304224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304230 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.304464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304240 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.304464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304249 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgbfp\" (UniqueName: \"kubernetes.io/projected/6954d991-78f1-4807-8ea7-db4ef795d2a7-kube-api-access-kgbfp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.304464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.304259 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/6954d991-78f1-4807-8ea7-db4ef795d2a7-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.514641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.514605 5577 generic.go:356] "Generic (PLEG): container finished" podID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerID="b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5" exitCode=0 Oct 31 00:35:25.514790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.514666 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" Oct 31 00:35:25.514790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.514693 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" event={"ID":"6954d991-78f1-4807-8ea7-db4ef795d2a7","Type":"ContainerDied","Data":"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5"} Oct 31 00:35:25.514790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.514780 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md" event={"ID":"6954d991-78f1-4807-8ea7-db4ef795d2a7","Type":"ContainerDied","Data":"3087babf9ffe3e63373c4b6d4868db9ac36435045366b8c30b5a34f7ab8f6cf9"} Oct 31 00:35:25.515066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.514809 5577 scope.go:122] "RemoveContainer" containerID="b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5" Oct 31 00:35:25.517505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.517485 5577 generic.go:356] "Generic (PLEG): container finished" podID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerID="4b7830bb51e6e9f6e7952f59f80273ae9980ffe9e15dcfbc50cef27d89016f4e" exitCode=0 Oct 31 00:35:25.517606 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.517529 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerDied","Data":"4b7830bb51e6e9f6e7952f59f80273ae9980ffe9e15dcfbc50cef27d89016f4e"} Oct 31 00:35:25.539319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.539290 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:25.565908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.565884 5577 scope.go:122] "RemoveContainer" containerID="b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5" Oct 31 00:35:25.566247 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:25.566227 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5\": container with ID starting with b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5 not found: ID does not exist" containerID="b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5" Oct 31 00:35:25.566340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.566255 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5"} err="failed to get container status \"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5\": rpc error: code = NotFound desc = could not find container \"b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5\": container with ID starting with b1ea0f6c3d60fbd5b457e3d04b7ada6b65c3cf696daf459a5fb0a16254b02cc5 not found: ID does not exist" Oct 31 00:35:25.580013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.579975 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:35:25.584370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.584345 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-574d99548c-h88md"] Oct 31 00:35:25.593665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.593638 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:35:25.680783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.680760 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:35:25.706676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706632 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.706873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706699 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.706873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706749 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.706873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706784 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.706873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706825 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.707087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706883 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.707087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.706925 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\" (UniqueName: \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\") pod \"a214686e-d4fc-467e-8ce7-c6858bc4c898\" (UID: \"a214686e-d4fc-467e-8ce7-c6858bc4c898\") " Oct 31 00:35:25.707833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.707622 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.707833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.707649 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.707833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.707797 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.709358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.709336 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "kube-api-access-cc68q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:25.709465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.709443 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.710094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.710067 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.714577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.714555 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant" pod "a214686e-d4fc-467e-8ce7-c6858bc4c898" (UID: "a214686e-d4fc-467e-8ce7-c6858bc4c898"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.807695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807656 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.807903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807756 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.807903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807787 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.807903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807817 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\" (UniqueName: \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.807903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807842 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.808131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.807994 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") pod \"2663b543-da9f-417d-b839-08f693ee8298\" (UID: \"2663b543-da9f-417d-b839-08f693ee8298\") " Oct 31 00:35:25.808227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808194 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.808282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808224 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc68q\" (UniqueName: \"kubernetes.io/projected/a214686e-d4fc-467e-8ce7-c6858bc4c898-kube-api-access-cc68q\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808245 5577 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-serving-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808260 5577 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tenant\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808275 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808290 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/a214686e-d4fc-467e-8ce7-c6858bc4c898-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808222 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:25.808470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808305 5577 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-rbac\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.808470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.808343 5577 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/a214686e-d4fc-467e-8ce7-c6858bc4c898-cabundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.810154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.810130 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "kube-api-access-6qvhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:25.810245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.810211 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.810371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.810346 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "tempo-data-science-tempostack-ingester-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:25.810844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.810825 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6" pod "2663b543-da9f-417d-b839-08f693ee8298" (UID: "2663b543-da9f-417d-b839-08f693ee8298"). InnerVolumeSpecName "pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909533 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909565 5577 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-storage-gcs-key\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909583 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909597 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2663b543-da9f-417d-b839-08f693ee8298-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909608 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qvhz\" (UniqueName: \"kubernetes.io/projected/2663b543-da9f-417d-b839-08f693ee8298-kube-api-access-6qvhz\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.909614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.909617 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ingester-mtls\" (UniqueName: \"kubernetes.io/secret/2663b543-da9f-417d-b839-08f693ee8298-tempo-data-science-tempostack-ingester-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:25.915234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.915208 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:35:25.915414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.915376 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/fake-gcs-server" podUID="75367784-658e-432c-9095-5ccb67fe1207" containerName="fake-gcs-server" containerID="cri-o://69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd" gracePeriod=30 Oct 31 00:35:25.918483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:25.918463 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:35:26.010154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.010121 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-1a0c5ca2-2766-4db0-b17f-4a8f45925f49\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0aac1093750eb10b6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-f05e1daa8a0e8f58357f7c0c4175de250b15de8ccf16d9c7ba80e381afe95735\"" Oct 31 00:35:26.154300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.154276 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:35:26.313735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.313677 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\" (UniqueName: \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\") pod \"75367784-658e-432c-9095-5ccb67fe1207\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " Oct 31 00:35:26.313929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.313744 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\" (UniqueName: \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\") pod \"75367784-658e-432c-9095-5ccb67fe1207\" (UID: \"75367784-658e-432c-9095-5ccb67fe1207\") " Oct 31 00:35:26.314080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.314056 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data" pod "75367784-658e-432c-9095-5ccb67fe1207" (UID: "75367784-658e-432c-9095-5ccb67fe1207"). InnerVolumeSpecName "data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:26.315909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.315889 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm" pod "75367784-658e-432c-9095-5ccb67fe1207" (UID: "75367784-658e-432c-9095-5ccb67fe1207"). InnerVolumeSpecName "kube-api-access-24gpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:26.414588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.414548 5577 reconciler_common.go:299] "Volume detached for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/75367784-658e-432c-9095-5ccb67fe1207-data\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:26.414588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.414581 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24gpm\" (UniqueName: \"kubernetes.io/projected/75367784-658e-432c-9095-5ccb67fe1207-kube-api-access-24gpm\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:26.438831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.438800 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:26.524287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.524259 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" Oct 31 00:35:26.524470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.524268 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt" event={"ID":"a214686e-d4fc-467e-8ce7-c6858bc4c898","Type":"ContainerDied","Data":"e889ddea1344b6c92a7b4419c085ba4229418a940fa92c0288fad4eb8b6858e3"} Oct 31 00:35:26.524470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.524421 5577 scope.go:122] "RemoveContainer" containerID="c0ba7c42bb07d557e46cfaf0b47c1f2f1cfcaadfd38bad071d0c4001ad41e258" Oct 31 00:35:26.525783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.525517 5577 generic.go:356] "Generic (PLEG): container finished" podID="75367784-658e-432c-9095-5ccb67fe1207" containerID="69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd" exitCode=0 Oct 31 00:35:26.525783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.525561 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/fake-gcs-server" Oct 31 00:35:26.525783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.525589 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"75367784-658e-432c-9095-5ccb67fe1207","Type":"ContainerDied","Data":"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd"} Oct 31 00:35:26.525783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.525627 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/fake-gcs-server" event={"ID":"75367784-658e-432c-9095-5ccb67fe1207","Type":"ContainerDied","Data":"cadde8b102f1fe140f3cbb4011a1683442cdee4f6aa27e85e1ed816514a2d520"} Oct 31 00:35:26.527000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.526980 5577 generic.go:356] "Generic (PLEG): container finished" podID="2663b543-da9f-417d-b839-08f693ee8298" containerID="f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d" exitCode=0 Oct 31 00:35:26.527093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.527032 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" Oct 31 00:35:26.527093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.527060 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2663b543-da9f-417d-b839-08f693ee8298","Type":"ContainerDied","Data":"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d"} Oct 31 00:35:26.527093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.527090 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0" event={"ID":"2663b543-da9f-417d-b839-08f693ee8298","Type":"ContainerDied","Data":"e6414d42de440b02abb2d9bd89b1f9e47bc7efc257b8b265bb37df644bac589b"} Oct 31 00:35:26.534899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.534820 5577 scope.go:122] "RemoveContainer" containerID="4b7830bb51e6e9f6e7952f59f80273ae9980ffe9e15dcfbc50cef27d89016f4e" Oct 31 00:35:26.542901 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.542883 5577 scope.go:122] "RemoveContainer" containerID="69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd" Oct 31 00:35:26.549478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.549453 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:35:26.550191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.550174 5577 scope.go:122] "RemoveContainer" containerID="69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd" Oct 31 00:35:26.550471 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:26.550453 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd\": container with ID starting with 69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd not found: ID does not exist" containerID="69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd" Oct 31 00:35:26.550521 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.550480 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd"} err="failed to get container status \"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd\": rpc error: code = NotFound desc = could not find container \"69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd\": container with ID starting with 69e9605ecb13e6d2d9d94171ee2fb6333fb45cd2ef27ffbf8fdaf5e27c76e2cd not found: ID does not exist" Oct 31 00:35:26.550521 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.550498 5577 scope.go:122] "RemoveContainer" containerID="f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d" Oct 31 00:35:26.554566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.554548 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/fake-gcs-server"] Oct 31 00:35:26.558511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.558496 5577 scope.go:122] "RemoveContainer" containerID="f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d" Oct 31 00:35:26.558775 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:26.558755 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d\": container with ID starting with f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d not found: ID does not exist" containerID="f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d" Oct 31 00:35:26.558837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.558784 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d"} err="failed to get container status \"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d\": rpc error: code = NotFound desc = could not find container \"f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d\": container with ID starting with f2ea40047e2e6589e101b29d122b60b003312da2a90437bbe15273953577f37d not found: ID does not exist" Oct 31 00:35:26.566998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.566948 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:35:26.573395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.573377 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-5b78f57785-7kbwt"] Oct 31 00:35:26.585226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.585206 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:35:26.591681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:26.591659 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-ingester-0"] Oct 31 00:35:27.129965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:27.129938 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2663b543-da9f-417d-b839-08f693ee8298" path="/var/lib/kubelet/pods/2663b543-da9f-417d-b839-08f693ee8298/volumes" Oct 31 00:35:27.130336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:27.130323 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" path="/var/lib/kubelet/pods/6954d991-78f1-4807-8ea7-db4ef795d2a7/volumes" Oct 31 00:35:27.130620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:27.130609 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75367784-658e-432c-9095-5ccb67fe1207" path="/var/lib/kubelet/pods/75367784-658e-432c-9095-5ccb67fe1207/volumes" Oct 31 00:35:27.131268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:27.131253 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" path="/var/lib/kubelet/pods/a214686e-d4fc-467e-8ce7-c6858bc4c898/volumes" Oct 31 00:35:28.061534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:28.061492 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:28.852044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:28.852010 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:28.977106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:28.977051 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:29.056673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.056643 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:29.057022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057008 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057025 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057041 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057047 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057053 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057058 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057064 5577 cpu_manager.go:414] "containerMap: removing container" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" Oct 31 00:35:29.057074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057068 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057086 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d775bbe0-668b-4a27-ae5c-18a28bf85470" containerName="tempo" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057090 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d775bbe0-668b-4a27-ae5c-18a28bf85470" containerName="tempo" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057098 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b6f20fb0-fda1-4580-b4da-341d02b65a87" containerName="perses" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f20fb0-fda1-4580-b4da-341d02b65a87" containerName="perses" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057114 5577 cpu_manager.go:414] "containerMap: removing container" podUID="75367784-658e-432c-9095-5ccb67fe1207" containerName="fake-gcs-server" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057118 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75367784-658e-432c-9095-5ccb67fe1207" containerName="fake-gcs-server" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057171 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d775bbe0-668b-4a27-ae5c-18a28bf85470" containerName="tempo" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057180 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f20fb0-fda1-4580-b4da-341d02b65a87" containerName="perses" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057189 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="75367784-658e-432c-9095-5ccb67fe1207" containerName="fake-gcs-server" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057200 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2663b543-da9f-417d-b839-08f693ee8298" containerName="tempo" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057207 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057212 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a214686e-d4fc-467e-8ce7-c6858bc4c898" containerName="tempo-gateway-opa" Oct 31 00:35:29.057268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.057217 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="6954d991-78f1-4807-8ea7-db4ef795d2a7" containerName="tempo" Oct 31 00:35:29.061916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.061892 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.071555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.071529 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:35:29.071948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.071907 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:35:29.072180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.072156 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 31 00:35:29.073275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.073253 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:29.099284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.099218 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:29.102920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.102853 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.114952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.114922 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:29.126023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.125987 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:29.170692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.170662 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:35:29.170897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.170784 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:35:29.240675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.240638 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.240870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.240687 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxm9h\" (UniqueName: \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.240870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.240803 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxgv7\" (UniqueName: \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.241012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.240888 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.241012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.240971 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.241129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.241009 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.241194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.241177 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.248197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.248166 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:29.292594 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:29.292561 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:35:29.341754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341717 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxm9h\" (UniqueName: \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.341920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341762 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxgv7\" (UniqueName: \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.341920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341799 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.341920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341836 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.341920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341866 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.342136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.341996 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.342136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.342075 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.342389 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.342366 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.342552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.342526 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.342672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.342650 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.344535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.344514 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.344535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.344524 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.354121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.354067 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxgv7\" (UniqueName: \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\") pod \"data-science-collector-collector-0\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.354214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.354128 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxm9h\" (UniqueName: \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:29.369056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.369022 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:29.380177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.380154 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:29.659738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.659641 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:29.663025 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:29.663000 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c73d6c_d93e_4c36_bc0d_53f71e609beb.slice/crio-27a035c8513674584593f643204d748a1445486e48014c38fe517d516a83ac8b WatchSource:0}: Error finding container 27a035c8513674584593f643204d748a1445486e48014c38fe517d516a83ac8b: Status 404 returned error can't find the container with id 27a035c8513674584593f643204d748a1445486e48014c38fe517d516a83ac8b Oct 31 00:35:29.665204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.665169 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:29.665287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:29.665227 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:30.024769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.024730 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:30.028429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.028412 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.032258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.032231 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-rjls8\"" Oct 31 00:35:30.032829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.032810 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:35:30.033158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033142 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:35:30.033277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033266 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:35:30.033330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033311 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:35:30.033589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033570 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:35:30.033772 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033624 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:35:30.033772 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.033695 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:35:30.045475 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.045453 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:30.096371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.096336 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:30.149530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149497 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149542 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149561 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx72\" (UniqueName: \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149738 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149793 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149817 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149886 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.149956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.149923 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.226626 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.226590 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:30.250667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250634 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250688 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250752 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250775 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx72\" (UniqueName: \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250818 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250856 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.250923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.250893 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.251257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.251038 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.251876 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.251852 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.251980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.251888 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.252093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.252066 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.252138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.252109 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.253832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.253809 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.254278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.254260 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.254552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.254529 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.263414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.263388 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx72\" (UniqueName: \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:30.548870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.548820 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"14c73d6c-d93e-4c36-bc0d-53f71e609beb","Type":"ContainerStarted","Data":"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe"} Oct 31 00:35:30.549029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.548890 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"14c73d6c-d93e-4c36-bc0d-53f71e609beb","Type":"ContainerStarted","Data":"27a035c8513674584593f643204d748a1445486e48014c38fe517d516a83ac8b"} Oct 31 00:35:30.568925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:30.568689 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.568671949 podStartE2EDuration="1.568671949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:30.567921629 +0000 UTC m=+2169.559581224" watchObservedRunningTime="2026-10-31 00:35:30.568671949 +0000 UTC m=+2169.560331534" Oct 31 00:35:31.053636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:31.053601 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:31.067624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:31.067594 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:36.101334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.101294 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.101734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.101354 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.108738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.108699 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.108830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.108737 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.201725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.201676 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.201879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.201757 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.201879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.201825 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.201879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.201875 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.205034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.205015 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.205108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.205062 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.297524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.297495 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.298412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.298385 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.314205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.314180 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.316621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.316599 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.344748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.344696 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:36.619035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.618964 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:36.627744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.627609 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:36.630863 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:36.630839 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5a22e5_addf_45fb_9597_45f4f42f733c.slice/crio-17746b3a3cb524948c5ff0283dc378be923ca4e1543da24680f2d25f7a665af6 WatchSource:0}: Error finding container 17746b3a3cb524948c5ff0283dc378be923ca4e1543da24680f2d25f7a665af6: Status 404 returned error can't find the container with id 17746b3a3cb524948c5ff0283dc378be923ca4e1543da24680f2d25f7a665af6 Oct 31 00:35:36.895216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:36.895150 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:36.898046 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:36.898021 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09772d38_384d_4c75_b9ed_ac00e2136020.slice/crio-212113efcd181a879724068a56a14ac58479d8ac66857b1b7a3f8ac986fc2291 WatchSource:0}: Error finding container 212113efcd181a879724068a56a14ac58479d8ac66857b1b7a3f8ac986fc2291: Status 404 returned error can't find the container with id 212113efcd181a879724068a56a14ac58479d8ac66857b1b7a3f8ac986fc2291 Oct 31 00:35:37.584620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.584586 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"09772d38-384d-4c75-b9ed-ac00e2136020","Type":"ContainerStarted","Data":"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867"} Oct 31 00:35:37.585135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.584631 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"09772d38-384d-4c75-b9ed-ac00e2136020","Type":"ContainerStarted","Data":"212113efcd181a879724068a56a14ac58479d8ac66857b1b7a3f8ac986fc2291"} Oct 31 00:35:37.585135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.584725 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:37.587047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587025 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerStarted","Data":"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250"} Oct 31 00:35:37.587175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587071 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerStarted","Data":"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17"} Oct 31 00:35:37.587175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587095 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerStarted","Data":"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b"} Oct 31 00:35:37.587175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587118 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerStarted","Data":"17746b3a3cb524948c5ff0283dc378be923ca4e1543da24680f2d25f7a665af6"} Oct 31 00:35:37.587175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587145 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:37.587175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.587169 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:37.590279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.590259 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:37.596392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.596374 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:37.604873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.604144 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=8.604128471 podStartE2EDuration="8.604128471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:35:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:37.60324738 +0000 UTC m=+2176.594906965" watchObservedRunningTime="2026-10-31 00:35:37.604128471 +0000 UTC m=+2176.595788056" Oct 31 00:35:37.628243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:37.628194 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=7.62817892 podStartE2EDuration="7.62817892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:35:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:37.627584259 +0000 UTC m=+2176.619243867" watchObservedRunningTime="2026-10-31 00:35:37.62817892 +0000 UTC m=+2176.619838569" Oct 31 00:35:38.592167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:38.592124 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:41.053473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:41.053432 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:41.166247 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:41.166199 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:41.384137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:41.384045 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:41.530104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:41.530075 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:44.905909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:44.905651 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:44.906378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:44.905924 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" containerName="otc-container" containerID="cri-o://87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe" gracePeriod=30 Oct 31 00:35:44.961647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:44.961616 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:44.961820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:44.961796 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="09772d38-384d-4c75-b9ed-ac00e2136020" containerName="perses" containerID="cri-o://a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867" gracePeriod=30 Oct 31 00:35:44.989539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:44.989506 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:45.017071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.017038 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:45.017313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.017243 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo" containerID="cri-o://e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b" gracePeriod=30 Oct 31 00:35:45.017411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.017304 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway-opa" containerID="cri-o://bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250" gracePeriod=30 Oct 31 00:35:45.017411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.017344 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway" containerID="cri-o://bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17" gracePeriod=30 Oct 31 00:35:45.184556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.184357 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:45.237746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.237725 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:45.256958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.256939 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:45.282085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.282048 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\") pod \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " Oct 31 00:35:45.282245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.282109 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\") pod \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " Oct 31 00:35:45.282245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.282136 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\") pod \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " Oct 31 00:35:45.282245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.282181 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\" (UniqueName: \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\") pod \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\" (UID: \"14c73d6c-d93e-4c36-bc0d-53f71e609beb\") " Oct 31 00:35:45.282525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.282500 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal" pod "14c73d6c-d93e-4c36-bc0d-53f71e609beb" (UID: "14c73d6c-d93e-4c36-bc0d-53f71e609beb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:45.284406 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.284377 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs" pod "14c73d6c-d93e-4c36-bc0d-53f71e609beb" (UID: "14c73d6c-d93e-4c36-bc0d-53f71e609beb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:45.284519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.284408 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs" pod "14c73d6c-d93e-4c36-bc0d-53f71e609beb" (UID: "14c73d6c-d93e-4c36-bc0d-53f71e609beb"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:45.284519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.284429 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7" pod "14c73d6c-d93e-4c36-bc0d-53f71e609beb" (UID: "14c73d6c-d93e-4c36-bc0d-53f71e609beb"). InnerVolumeSpecName "kube-api-access-xxgv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:45.307889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.307858 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:45.383229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383204 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\" (UniqueName: \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\") pod \"09772d38-384d-4c75-b9ed-ac00e2136020\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " Oct 31 00:35:45.383343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383300 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"09772d38-384d-4c75-b9ed-ac00e2136020\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " Oct 31 00:35:45.383343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383334 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\" (UniqueName: \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\") pod \"09772d38-384d-4c75-b9ed-ac00e2136020\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " Oct 31 00:35:45.383469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383371 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\" (UniqueName: \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\") pod \"09772d38-384d-4c75-b9ed-ac00e2136020\" (UID: \"09772d38-384d-4c75-b9ed-ac00e2136020\") " Oct 31 00:35:45.383551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383537 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.383605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383552 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/14c73d6c-d93e-4c36-bc0d-53f71e609beb-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.383605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383549 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config" pod "09772d38-384d-4c75-b9ed-ac00e2136020" (UID: "09772d38-384d-4c75-b9ed-ac00e2136020"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:45.383605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383562 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxgv7\" (UniqueName: \"kubernetes.io/projected/14c73d6c-d93e-4c36-bc0d-53f71e609beb-kube-api-access-xxgv7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.383772 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.383592 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/14c73d6c-d93e-4c36-bc0d-53f71e609beb-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.385620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.385592 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h" pod "09772d38-384d-4c75-b9ed-ac00e2136020" (UID: "09772d38-384d-4c75-b9ed-ac00e2136020"). InnerVolumeSpecName "kube-api-access-fxm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:45.386223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.386206 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "09772d38-384d-4c75-b9ed-ac00e2136020" (UID: "09772d38-384d-4c75-b9ed-ac00e2136020"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:35:45.484661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.484593 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/09772d38-384d-4c75-b9ed-ac00e2136020-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.484661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.484648 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:35:45.484833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.484666 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxm9h\" (UniqueName: \"kubernetes.io/projected/09772d38-384d-4c75-b9ed-ac00e2136020-kube-api-access-fxm9h\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.495267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.495247 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:35:45.505395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.505371 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins" pod "09772d38-384d-4c75-b9ed-ac00e2136020" (UID: "09772d38-384d-4c75-b9ed-ac00e2136020"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:45.585492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.585463 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:35:45.585492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.585487 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/09772d38-384d-4c75-b9ed-ac00e2136020-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:45.623459 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.623429 5577 generic.go:356] "Generic (PLEG): container finished" podID="09772d38-384d-4c75-b9ed-ac00e2136020" containerID="a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867" exitCode=0 Oct 31 00:35:45.623569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.623480 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:45.623569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.623511 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"09772d38-384d-4c75-b9ed-ac00e2136020","Type":"ContainerDied","Data":"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867"} Oct 31 00:35:45.623569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.623541 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"09772d38-384d-4c75-b9ed-ac00e2136020","Type":"ContainerDied","Data":"212113efcd181a879724068a56a14ac58479d8ac66857b1b7a3f8ac986fc2291"} Oct 31 00:35:45.623569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.623556 5577 scope.go:122] "RemoveContainer" containerID="a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867" Oct 31 00:35:45.625874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.625850 5577 generic.go:356] "Generic (PLEG): container finished" podID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerID="bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250" exitCode=0 Oct 31 00:35:45.625874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.625874 5577 generic.go:356] "Generic (PLEG): container finished" podID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerID="e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b" exitCode=0 Oct 31 00:35:45.626024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.625923 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerDied","Data":"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250"} Oct 31 00:35:45.626024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.625963 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerDied","Data":"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b"} Oct 31 00:35:45.627321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.627301 5577 generic.go:356] "Generic (PLEG): container finished" podID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" containerID="87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe" exitCode=0 Oct 31 00:35:45.627394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.627348 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 31 00:35:45.627442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.627339 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"14c73d6c-d93e-4c36-bc0d-53f71e609beb","Type":"ContainerDied","Data":"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe"} Oct 31 00:35:45.627489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.627474 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"14c73d6c-d93e-4c36-bc0d-53f71e609beb","Type":"ContainerDied","Data":"27a035c8513674584593f643204d748a1445486e48014c38fe517d516a83ac8b"} Oct 31 00:35:45.632646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.632512 5577 scope.go:122] "RemoveContainer" containerID="a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867" Oct 31 00:35:45.632928 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:45.632899 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867\": container with ID starting with a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867 not found: ID does not exist" containerID="a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867" Oct 31 00:35:45.633022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.632938 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867"} err="failed to get container status \"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867\": rpc error: code = NotFound desc = could not find container \"a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867\": container with ID starting with a13b359627d825f24c1786fd27a96f678168e05a79954e85e7ab4f887cccf867 not found: ID does not exist" Oct 31 00:35:45.633022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.632959 5577 scope.go:122] "RemoveContainer" containerID="87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe" Oct 31 00:35:45.641445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.641429 5577 scope.go:122] "RemoveContainer" containerID="87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe" Oct 31 00:35:45.641701 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:45.641684 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe\": container with ID starting with 87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe not found: ID does not exist" containerID="87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe" Oct 31 00:35:45.641771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.641729 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe"} err="failed to get container status \"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe\": rpc error: code = NotFound desc = could not find container \"87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe\": container with ID starting with 87b8cde7a36325f6c179721436406a7d415f35b7d87c25841d03376eb48d9dbe not found: ID does not exist" Oct 31 00:35:45.646744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.646695 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:45.651184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.651163 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:45.662952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.662934 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:45.665599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:45.665580 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 31 00:35:46.262062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.262040 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:46.391381 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391350 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391410 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391440 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391477 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\" (UniqueName: \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391516 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391546 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391577 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391609 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.391676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391637 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"2a5a22e5-addf-45fb-9597-45f4f42f733c\" (UID: \"2a5a22e5-addf-45fb-9597-45f4f42f733c\") " Oct 31 00:35:46.392180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.391989 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:46.392180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.392013 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:46.392180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.392100 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-rbac\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.392180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.392119 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.392180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.392155 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:46.392505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.392481 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:46.393925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.393903 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "kube-api-access-jbx72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:46.394038 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.393991 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:46.394566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.394540 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:46.394659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.394565 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:46.394659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.394592 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "2a5a22e5-addf-45fb-9597-45f4f42f733c" (UID: "2a5a22e5-addf-45fb-9597-45f4f42f733c"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:35:46.493079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493013 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbx72\" (UniqueName: \"kubernetes.io/projected/2a5a22e5-addf-45fb-9597-45f4f42f733c-kube-api-access-jbx72\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.493079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493035 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.493079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493047 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.493079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493058 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-gateway-tenants\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.493079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493067 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.493328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493093 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:35:46.493328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.493104 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/2a5a22e5-addf-45fb-9597-45f4f42f733c-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:46.502341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.502325 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:35:46.593878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.593854 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" Oct 31 00:35:46.634078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.634053 5577 generic.go:356] "Generic (PLEG): container finished" podID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerID="bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17" exitCode=0 Oct 31 00:35:46.634178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.634107 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:35:46.634178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.634138 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerDied","Data":"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17"} Oct 31 00:35:46.634295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.634193 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"2a5a22e5-addf-45fb-9597-45f4f42f733c","Type":"ContainerDied","Data":"17746b3a3cb524948c5ff0283dc378be923ca4e1543da24680f2d25f7a665af6"} Oct 31 00:35:46.634295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.634217 5577 scope.go:122] "RemoveContainer" containerID="bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250" Oct 31 00:35:46.642732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.642696 5577 scope.go:122] "RemoveContainer" containerID="bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17" Oct 31 00:35:46.650282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.650267 5577 scope.go:122] "RemoveContainer" containerID="e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b" Oct 31 00:35:46.657162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657142 5577 scope.go:122] "RemoveContainer" containerID="bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250" Oct 31 00:35:46.657428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:46.657404 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250\": container with ID starting with bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250 not found: ID does not exist" containerID="bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250" Oct 31 00:35:46.657511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657435 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250"} err="failed to get container status \"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250\": rpc error: code = NotFound desc = could not find container \"bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250\": container with ID starting with bc0f2ebb82247ab7932c995e7108c30c380711cdf3df0f0a48f1ca41037a3250 not found: ID does not exist" Oct 31 00:35:46.657511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657461 5577 scope.go:122] "RemoveContainer" containerID="bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17" Oct 31 00:35:46.657736 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:46.657698 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17\": container with ID starting with bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17 not found: ID does not exist" containerID="bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17" Oct 31 00:35:46.657785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657739 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17"} err="failed to get container status \"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17\": rpc error: code = NotFound desc = could not find container \"bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17\": container with ID starting with bf64481468423e3df9a8341ac5825e5770439a1fd96f02a4fde0d86f17a65c17 not found: ID does not exist" Oct 31 00:35:46.657785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657751 5577 scope.go:122] "RemoveContainer" containerID="e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b" Oct 31 00:35:46.657990 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:46.657972 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b\": container with ID starting with e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b not found: ID does not exist" containerID="e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b" Oct 31 00:35:46.658031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.657996 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b"} err="failed to get container status \"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b\": rpc error: code = NotFound desc = could not find container \"e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b\": container with ID starting with e9ffae1d5422bd2db51f8769412cbc0c67850175e20d4e3b698f33d8cfb7e44b not found: ID does not exist" Oct 31 00:35:46.667271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.667251 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:46.672839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:46.672821 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:35:47.129493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:47.129459 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09772d38-384d-4c75-b9ed-ac00e2136020" path="/var/lib/kubelet/pods/09772d38-384d-4c75-b9ed-ac00e2136020/volumes" Oct 31 00:35:47.129916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:47.129899 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" path="/var/lib/kubelet/pods/14c73d6c-d93e-4c36-bc0d-53f71e609beb/volumes" Oct 31 00:35:47.130271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:47.130258 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" path="/var/lib/kubelet/pods/2a5a22e5-addf-45fb-9597-45f4f42f733c/volumes" Oct 31 00:35:51.053823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:51.053788 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:51.054258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:51.053915 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:35:51.054258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:51.054243 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:35:51.054361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:51.054332 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:35:53.333650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.333609 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:53.432437 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432407 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:35:53.432749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432737 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo" Oct 31 00:35:53.432787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432750 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo" Oct 31 00:35:53.432787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432763 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway" Oct 31 00:35:53.432787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432768 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway" Oct 31 00:35:53.432787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432776 5577 cpu_manager.go:414] "containerMap: removing container" podUID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" containerName="otc-container" Oct 31 00:35:53.432787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432781 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" containerName="otc-container" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432790 5577 cpu_manager.go:414] "containerMap: removing container" podUID="09772d38-384d-4c75-b9ed-ac00e2136020" containerName="perses" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432795 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09772d38-384d-4c75-b9ed-ac00e2136020" containerName="perses" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432803 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway-opa" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432807 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway-opa" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432872 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="09772d38-384d-4c75-b9ed-ac00e2136020" containerName="perses" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432880 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway-opa" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432890 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432896 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5a22e5-addf-45fb-9597-45f4f42f733c" containerName="tempo-gateway" Oct 31 00:35:53.432951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.432902 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="14c73d6c-d93e-4c36-bc0d-53f71e609beb" containerName="otc-container" Oct 31 00:35:53.437934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.437912 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.447087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.447064 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:35:53.452325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.452301 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:35:53.455774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.455756 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.456289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.456266 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:35:53.456420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.456409 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-b7575\"" Oct 31 00:35:53.468653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.468624 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:35:53.487128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.487097 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-mt978\"" Oct 31 00:35:53.487293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.487232 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:35:53.488403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.488374 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:53.518197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.518169 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:35:53.518197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.518195 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:35:53.518349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.518280 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:35:53.518349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.518316 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:35:53.557037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557008 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7dn\" (UniqueName: \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.557150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557060 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jjtc\" (UniqueName: \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.557190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557150 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.557190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557182 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.557293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557211 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.557346 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557304 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.557402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557347 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.557452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.557434 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.617970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.617912 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:53.657840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.657811 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.657998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.657863 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7dn\" (UniqueName: \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.657998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.657917 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jjtc\" (UniqueName: \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.657998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.657952 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.657998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.657979 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.658284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658012 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.658284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658035 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.658284 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658065 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.658569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658543 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.658634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658593 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.658792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658771 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.658892 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.658874 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.661025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.661001 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.661463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.661407 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.674052 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.674025 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jjtc\" (UniqueName: \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bzrr7\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.674746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.674695 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7dn\" (UniqueName: \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-zw2h6\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.741507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.741472 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:53.757318 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.757288 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:35:53.775489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.775467 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:35:53.951614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.951526 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:53.956140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.956119 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:53.968203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.968141 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:53.972875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.972852 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:35:53.977467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.977440 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:53.981485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.981465 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:53.990606 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:53.990586 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:54.024749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.024697 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:54.037808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.037777 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:35:54.044453 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:54.044427 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f8fa61_d4dd_4a65_9015_249bd2067703.slice/crio-e1d4f0054cbe296c05ce1b770a93bf21420854ba8bb64714cc144df6e93fbc45 WatchSource:0}: Error finding container e1d4f0054cbe296c05ce1b770a93bf21420854ba8bb64714cc144df6e93fbc45: Status 404 returned error can't find the container with id e1d4f0054cbe296c05ce1b770a93bf21420854ba8bb64714cc144df6e93fbc45 Oct 31 00:35:54.044830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.044739 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:35:54.044830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.044773 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:35:54.044992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.044932 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:35:54.044992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.044938 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:35:54.044992 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.044949 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:35:54.045136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.045032 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:35:54.045253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.045144 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:35:54.060033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060013 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.060141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060046 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97dbf\" (UniqueName: \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.060141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060086 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.060141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060129 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.060319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060156 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.060319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060179 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.060319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060199 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.060319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060236 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.060319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.060270 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsnkz\" (UniqueName: \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.069184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.069161 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:35:54.073698 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:54.073678 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99107d2_526a_494e_accd_e32ca14a5507.slice/crio-45b1fc06aed0a763338d2313158a0b1cefbb63d8821086877bd4eea41f196316 WatchSource:0}: Error finding container 45b1fc06aed0a763338d2313158a0b1cefbb63d8821086877bd4eea41f196316: Status 404 returned error can't find the container with id 45b1fc06aed0a763338d2313158a0b1cefbb63d8821086877bd4eea41f196316 Oct 31 00:35:54.154128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.154099 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:54.160985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.160960 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161009 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97dbf\" (UniqueName: \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.161071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161057 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161106 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161137 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161174 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161205 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.161327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161252 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.161327 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:54.161272 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:35:54.161327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161287 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsnkz\" (UniqueName: \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.161480 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:54.161334 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs podName:0a4e00ed-78cc-49de-886c-071b60260739 nodeName:}" failed. No retries permitted until 2026-10-31 00:35:54.661313376 +0000 UTC m=+2193.652972952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "0a4e00ed-78cc-49de-886c-071b60260739") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:35:54.161584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161566 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.161936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.161915 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.162208 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.162188 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.164014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.163984 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.164135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.164119 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.164193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.164172 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.177941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.174698 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsnkz\" (UniqueName: \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.178402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.178375 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97dbf\" (UniqueName: \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:54.489154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.489130 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:35:54.510825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.510662 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:54.519291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.519267 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565126 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\" (UniqueName: \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565195 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565246 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565287 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565315 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.565749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.565356 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\") pod \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\" (UID: \"be4187c9-ce3c-4035-83ed-6cafdffb3a69\") " Oct 31 00:35:54.566850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.566562 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:54.567467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.567446 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:54.567648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.567618 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:54.571420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.571396 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:54.571558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.571520 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:54.571642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.571554 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85" pod "be4187c9-ce3c-4035-83ed-6cafdffb3a69" (UID: "be4187c9-ce3c-4035-83ed-6cafdffb3a69"). InnerVolumeSpecName "kube-api-access-gfp85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:54.666371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666333 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666393 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\" (UniqueName: \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666473 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666531 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666557 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666600 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\") pod \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\" (UID: \"82ecd7df-e24c-41cb-bebf-3ffaf8063024\") " Oct 31 00:35:54.666831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666818 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.666936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666879 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfp85\" (UniqueName: \"kubernetes.io/projected/be4187c9-ce3c-4035-83ed-6cafdffb3a69-kube-api-access-gfp85\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.666936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666896 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.666936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666912 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.666936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666927 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.667137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666940 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/be4187c9-ce3c-4035-83ed-6cafdffb3a69-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.667137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.666954 5577 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/be4187c9-ce3c-4035-83ed-6cafdffb3a69-storage-gcs-key\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.667636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.667511 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:54.667859 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.667827 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:35:54.668363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.668336 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:35:54.670626 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.670556 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:54.670745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.670687 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.670858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.670839 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "kube-api-access-xldcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:35:54.670948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.670934 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key" pod "82ecd7df-e24c-41cb-bebf-3ffaf8063024" (UID: "82ecd7df-e24c-41cb-bebf-3ffaf8063024"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:35:54.679277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.679159 5577 generic.go:356] "Generic (PLEG): container finished" podID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerID="44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2" exitCode=137 Oct 31 00:35:54.679277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.679214 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" Oct 31 00:35:54.679277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.679207 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" event={"ID":"be4187c9-ce3c-4035-83ed-6cafdffb3a69","Type":"ContainerDied","Data":"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2"} Oct 31 00:35:54.679485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.679371 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd" event={"ID":"be4187c9-ce3c-4035-83ed-6cafdffb3a69","Type":"ContainerDied","Data":"4bd1b5eb3b3e4136117b84558ff5fd775194270b5b61bb1dfbfe2c42540b6f02"} Oct 31 00:35:54.679485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.679414 5577 scope.go:122] "RemoveContainer" containerID="44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2" Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.681414 5577 generic.go:356] "Generic (PLEG): container finished" podID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerID="1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2" exitCode=137 Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.681488 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" event={"ID":"82ecd7df-e24c-41cb-bebf-3ffaf8063024","Type":"ContainerDied","Data":"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2"} Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.681536 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" event={"ID":"82ecd7df-e24c-41cb-bebf-3ffaf8063024","Type":"ContainerDied","Data":"edf9bfedfe1f46bde567ae2b28d325d47b24e46fdd91fa4b80e3699b6bec558e"} Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.681616 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb" Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.689456 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerStarted","Data":"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841"} Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.689515 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerStarted","Data":"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d"} Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.689543 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerStarted","Data":"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3"} Oct 31 00:35:54.689751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.689581 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerStarted","Data":"45b1fc06aed0a763338d2313158a0b1cefbb63d8821086877bd4eea41f196316"} Oct 31 00:35:54.692150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.692115 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerStarted","Data":"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8"} Oct 31 00:35:54.692278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.692166 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerStarted","Data":"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921"} Oct 31 00:35:54.692278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.692198 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerStarted","Data":"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72"} Oct 31 00:35:54.692278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.692224 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerStarted","Data":"e1d4f0054cbe296c05ce1b770a93bf21420854ba8bb64714cc144df6e93fbc45"} Oct 31 00:35:54.707102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.707081 5577 scope.go:122] "RemoveContainer" containerID="44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2" Oct 31 00:35:54.707395 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:54.707369 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2\": container with ID starting with 44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2 not found: ID does not exist" containerID="44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2" Oct 31 00:35:54.707486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.707405 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2"} err="failed to get container status \"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2\": rpc error: code = NotFound desc = could not find container \"44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2\": container with ID starting with 44398090cd81c62ebe715ab29e85fb4fb2ce19519df3d21bfc26d520da46e6b2 not found: ID does not exist" Oct 31 00:35:54.707486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.707427 5577 scope.go:122] "RemoveContainer" containerID="1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2" Oct 31 00:35:54.716309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.716291 5577 scope.go:122] "RemoveContainer" containerID="1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2" Oct 31 00:35:54.716608 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:54.716584 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2\": container with ID starting with 1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2 not found: ID does not exist" containerID="1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2" Oct 31 00:35:54.716749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.716612 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2"} err="failed to get container status \"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2\": rpc error: code = NotFound desc = could not find container \"1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2\": container with ID starting with 1f0ca7ba1fc15baa5520ae200e3d2d9936e84b2062cd9495a5adba2215ca2ae2 not found: ID does not exist" Oct 31 00:35:54.729540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.729496 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" podStartSLOduration=1.729484039 podStartE2EDuration="1.729484039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:54.728837784 +0000 UTC m=+2193.720497368" watchObservedRunningTime="2026-10-31 00:35:54.729484039 +0000 UTC m=+2193.721143622" Oct 31 00:35:54.750005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.749953 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" podStartSLOduration=1.7499339470000002 podStartE2EDuration="1.749933947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:54.748938577 +0000 UTC m=+2193.740598190" watchObservedRunningTime="2026-10-31 00:35:54.749933947 +0000 UTC m=+2193.741593536" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767604 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767637 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767655 5577 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-storage-gcs-key\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767670 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-tmp-storage\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767684 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/82ecd7df-e24c-41cb-bebf-3ffaf8063024-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.768053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.767698 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xldcl\" (UniqueName: \"kubernetes.io/projected/82ecd7df-e24c-41cb-bebf-3ffaf8063024-kube-api-access-xldcl\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:35:54.770853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.770830 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:35:54.779888 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.779862 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-84dcb46bf9-t89sd"] Oct 31 00:35:54.791632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.791608 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:35:54.797232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.797212 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-68c8969988-qbhcb"] Oct 31 00:35:54.869989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.869963 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:35:54.904945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:54.904916 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:55.099490 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.099406 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:35:55.132141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.131260 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" path="/var/lib/kubelet/pods/82ecd7df-e24c-41cb-bebf-3ffaf8063024/volumes" Oct 31 00:35:55.132141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.131841 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" path="/var/lib/kubelet/pods/be4187c9-ce3c-4035-83ed-6cafdffb3a69/volumes" Oct 31 00:35:55.154269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.154244 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:35:55.157207 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:55.157175 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a4e00ed_78cc_49de_886c_071b60260739.slice/crio-e6a2bd491a30668b9d1d6de596b72ac7dc88324aecdbbe4f0cc3576a5cb81a12 WatchSource:0}: Error finding container e6a2bd491a30668b9d1d6de596b72ac7dc88324aecdbbe4f0cc3576a5cb81a12: Status 404 returned error can't find the container with id e6a2bd491a30668b9d1d6de596b72ac7dc88324aecdbbe4f0cc3576a5cb81a12 Oct 31 00:35:55.159775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.159743 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:55.159859 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.159805 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:35:55.699456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.699422 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a4e00ed-78cc-49de-886c-071b60260739","Type":"ContainerStarted","Data":"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e"} Oct 31 00:35:55.699914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.699488 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a4e00ed-78cc-49de-886c-071b60260739","Type":"ContainerStarted","Data":"e6a2bd491a30668b9d1d6de596b72ac7dc88324aecdbbe4f0cc3576a5cb81a12"} Oct 31 00:35:55.719384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:55.719343 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.7193322 podStartE2EDuration="2.7193322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:55.718388263 +0000 UTC m=+2194.710047871" watchObservedRunningTime="2026-10-31 00:35:55.7193322 +0000 UTC m=+2194.710991778" Oct 31 00:35:56.784617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.784583 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:56.791611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.791586 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:56.885736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.885680 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:56.885865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.885803 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:56.895288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.895266 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:56.986807 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:56.986772 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:57.005666 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.005641 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:57.294861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.294814 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:57.568863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.568785 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:35:57.572207 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:35:57.572182 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe23abc0_0155_4ed5_8c30_fd61fa5bde5e.slice/crio-86ac9315112a1cd51390b642d2d378a38be5da9639d864b4d02a929605439030 WatchSource:0}: Error finding container 86ac9315112a1cd51390b642d2d378a38be5da9639d864b4d02a929605439030: Status 404 returned error can't find the container with id 86ac9315112a1cd51390b642d2d378a38be5da9639d864b4d02a929605439030 Oct 31 00:35:57.711783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.711744 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"be23abc0-0155-4ed5-8c30-fd61fa5bde5e","Type":"ContainerStarted","Data":"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf"} Oct 31 00:35:57.711954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.711795 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"be23abc0-0155-4ed5-8c30-fd61fa5bde5e","Type":"ContainerStarted","Data":"86ac9315112a1cd51390b642d2d378a38be5da9639d864b4d02a929605439030"} Oct 31 00:35:57.711954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.711895 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:35:57.730555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:57.730503 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.730489897 podStartE2EDuration="4.730489897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:35:57.730021757 +0000 UTC m=+2196.721681342" watchObservedRunningTime="2026-10-31 00:35:57.730489897 +0000 UTC m=+2196.722149511" Oct 31 00:35:59.293901 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:35:59.293849 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:35:59.844367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:35:59.844335 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:00.056349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.056315 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:00.194730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.194636 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:00.411921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.411885 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:36:00.412342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412271 5577 cpu_manager.go:414] "containerMap: removing container" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" Oct 31 00:36:00.412342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412284 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" Oct 31 00:36:00.412342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412332 5577 cpu_manager.go:414] "containerMap: removing container" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" Oct 31 00:36:00.412342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412337 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" Oct 31 00:36:00.412494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412396 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="be4187c9-ce3c-4035-83ed-6cafdffb3a69" containerName="tempo" Oct 31 00:36:00.412494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.412406 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ecd7df-e24c-41cb-bebf-3ffaf8063024" containerName="tempo" Oct 31 00:36:00.435909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.435852 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:00.436419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.436198 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:36:00.436419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.436338 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.510950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.510898 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:36:00.511128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.511014 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-pn2f5\"" Oct 31 00:36:00.511128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.511027 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:36:00.511128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.511043 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:36:00.511128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.511051 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:36:00.517538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517506 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.517658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517628 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.517723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517670 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rph\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.517723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517693 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.517830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517734 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.517953 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.517932 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.518012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.518000 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.575437 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.575401 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:00.619416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619380 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619417 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619455 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619489 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619537 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619597 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619636 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rph\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.619842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.619835 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.622211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.622189 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.622513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.622488 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.623320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.623293 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.623414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.623300 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.623625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.623608 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.632382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.632359 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rph\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:00.759729 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:00.759677 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:01.053090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:01.053011 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:36:01.056063 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:36:01.056029 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e97df37_be64_4b72_a93b_9dbf940ca86b.slice/crio-3e561738106b7e11df3376e6c8b2cc840fe17799bec235238468526bc381ba52 WatchSource:0}: Error finding container 3e561738106b7e11df3376e6c8b2cc840fe17799bec235238468526bc381ba52: Status 404 returned error can't find the container with id 3e561738106b7e11df3376e6c8b2cc840fe17799bec235238468526bc381ba52 Oct 31 00:36:01.733703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:01.733624 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerStarted","Data":"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5"} Oct 31 00:36:01.733703 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:01.733680 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerStarted","Data":"3e561738106b7e11df3376e6c8b2cc840fe17799bec235238468526bc381ba52"} Oct 31 00:36:05.754620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:05.754586 5577 generic.go:356] "Generic (PLEG): container finished" podID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerID="5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5" exitCode=0 Oct 31 00:36:05.755099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:05.754640 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerDied","Data":"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5"} Oct 31 00:36:06.760894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:06.760864 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerStarted","Data":"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf"} Oct 31 00:36:06.761273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:06.760921 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerStarted","Data":"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1"} Oct 31 00:36:06.761273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:06.761097 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:06.791101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:06.791061 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.406815973 podStartE2EDuration="6.791050952s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384234979s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:06.789465513 +0000 UTC m=+2205.781125097" watchObservedRunningTime="2026-10-31 00:36:06.791050952 +0000 UTC m=+2205.782710532" Oct 31 00:36:08.720761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:08.720728 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:36:12.767533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:12.767502 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:36:26.501074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:26.501015 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:29.292005 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:36:29.291968 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:36:32.851868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:32.851837 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:32.985240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:32.985204 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.353115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:33.353080 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:33.353349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:33.353312 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0a4e00ed-78cc-49de-886c-071b60260739" containerName="otc-container" containerID="cri-o://30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e" gracePeriod=30 Oct 31 00:36:33.477925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:33.477887 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:33.607995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:33.607914 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:34.609754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.609730 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:34.707285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707211 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\" (UniqueName: \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707258 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707289 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707309 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707327 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707511 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707367 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\") pod \"0a4e00ed-78cc-49de-886c-071b60260739\" (UID: \"0a4e00ed-78cc-49de-886c-071b60260739\") " Oct 31 00:36:34.707742 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.707691 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:36:34.709882 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.709845 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:34.709998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.709909 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:34.709998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.709924 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:34.710121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.710064 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:36:34.710121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.710106 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz" pod "0a4e00ed-78cc-49de-886c-071b60260739" (UID: "0a4e00ed-78cc-49de-886c-071b60260739"). InnerVolumeSpecName "kube-api-access-tsnkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:36:34.808568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808530 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.808568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808559 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsnkz\" (UniqueName: \"kubernetes.io/projected/0a4e00ed-78cc-49de-886c-071b60260739-kube-api-access-tsnkz\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.808568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808572 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.808863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808580 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0a4e00ed-78cc-49de-886c-071b60260739-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.808863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808590 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.808863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.808598 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a4e00ed-78cc-49de-886c-071b60260739-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:36:34.887220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.887161 5577 generic.go:356] "Generic (PLEG): container finished" podID="0a4e00ed-78cc-49de-886c-071b60260739" containerID="30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e" exitCode=0 Oct 31 00:36:34.887220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.887209 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:34.887446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.887246 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a4e00ed-78cc-49de-886c-071b60260739","Type":"ContainerDied","Data":"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e"} Oct 31 00:36:34.887446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.887297 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0a4e00ed-78cc-49de-886c-071b60260739","Type":"ContainerDied","Data":"e6a2bd491a30668b9d1d6de596b72ac7dc88324aecdbbe4f0cc3576a5cb81a12"} Oct 31 00:36:34.887446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.887319 5577 scope.go:122] "RemoveContainer" containerID="30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e" Oct 31 00:36:34.895856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.895841 5577 scope.go:122] "RemoveContainer" containerID="30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e" Oct 31 00:36:34.896107 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:36:34.896090 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e\": container with ID starting with 30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e not found: ID does not exist" containerID="30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e" Oct 31 00:36:34.896165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.896115 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e"} err="failed to get container status \"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e\": rpc error: code = NotFound desc = could not find container \"30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e\": container with ID starting with 30954dfd43c551df5cca82fab715d6d80834cc38f731d3e23ebf751530dca77e not found: ID does not exist" Oct 31 00:36:34.909311 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.909287 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:34.915781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.915757 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:34.942745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.942721 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:34.943049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.943037 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0a4e00ed-78cc-49de-886c-071b60260739" containerName="otc-container" Oct 31 00:36:34.943091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.943050 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a4e00ed-78cc-49de-886c-071b60260739" containerName="otc-container" Oct 31 00:36:34.943125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.943107 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a4e00ed-78cc-49de-886c-071b60260739" containerName="otc-container" Oct 31 00:36:34.947417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.947403 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:34.951683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.951664 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:36:34.951683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.951676 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:36:34.951683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.951667 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:36:34.951864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.951739 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:36:34.951864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.951778 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:36:34.952258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.952243 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:36:34.959151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:34.959090 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:35.009790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.009759 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.009928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.009796 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.009928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.009818 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.009928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.009881 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.010092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.009981 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d854s\" (UniqueName: \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.010092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.010025 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110301 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110335 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110354 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110369 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110395 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d854s\" (UniqueName: \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.110470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.110448 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.111123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.111077 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.113544 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.113516 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.113636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.113543 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.113636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.113584 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.113636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.113600 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.122823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.122801 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d854s\" (UniqueName: \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\") pod \"data-science-collector-collector-1\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.130091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.130068 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a4e00ed-78cc-49de-886c-071b60260739" path="/var/lib/kubelet/pods/0a4e00ed-78cc-49de-886c-071b60260739/volumes" Oct 31 00:36:35.259853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.259823 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:36:35.534942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.534499 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:36:35.537389 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:36:35.537359 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod829c41c6_7add_4cd1_a56d_811b87b13e7e.slice/crio-097552cdf878886409eff62dc65242d6f6c3ff587a9b89b09c2cdc6a8661eb64 WatchSource:0}: Error finding container 097552cdf878886409eff62dc65242d6f6c3ff587a9b89b09c2cdc6a8661eb64: Status 404 returned error can't find the container with id 097552cdf878886409eff62dc65242d6f6c3ff587a9b89b09c2cdc6a8661eb64 Oct 31 00:36:35.539648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.539617 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:36:35.539762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.539673 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:36:35.574183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.574161 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38"] Oct 31 00:36:35.578996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.578978 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" Oct 31 00:36:35.584273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.584247 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38"] Oct 31 00:36:35.591677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.591657 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" Oct 31 00:36:35.864303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.864227 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38"] Oct 31 00:36:35.867767 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:36:35.867735 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9bda4c16_8ea5_4e7e_adda_8215696d6ce4.slice/crio-43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c WatchSource:0}: Error finding container 43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c: Status 404 returned error can't find the container with id 43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c Oct 31 00:36:35.869854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.869836 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:36:35.893601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.893569 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"829c41c6-7add-4cd1-a56d-811b87b13e7e","Type":"ContainerStarted","Data":"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a"} Oct 31 00:36:35.893702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.893629 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"829c41c6-7add-4cd1-a56d-811b87b13e7e","Type":"ContainerStarted","Data":"097552cdf878886409eff62dc65242d6f6c3ff587a9b89b09c2cdc6a8661eb64"} Oct 31 00:36:35.895829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.895806 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" event={"ID":"9bda4c16-8ea5-4e7e-adda-8215696d6ce4","Type":"ContainerStarted","Data":"43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c"} Oct 31 00:36:35.919254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:35.919202 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.919185863 podStartE2EDuration="1.919185863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:35.917827514 +0000 UTC m=+2234.909487098" watchObservedRunningTime="2026-10-31 00:36:35.919185863 +0000 UTC m=+2234.910845448" Oct 31 00:36:36.902044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:36.902014 5577 generic.go:356] "Generic (PLEG): container finished" podID="9bda4c16-8ea5-4e7e-adda-8215696d6ce4" containerID="6c2d0eb193e8acdea9aaa559b9c2e031fc4d30703ea7ba577331bf2e3be486f5" exitCode=0 Oct 31 00:36:36.902416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:36.902102 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" event={"ID":"9bda4c16-8ea5-4e7e-adda-8215696d6ce4","Type":"ContainerDied","Data":"6c2d0eb193e8acdea9aaa559b9c2e031fc4d30703ea7ba577331bf2e3be486f5"} Oct 31 00:36:38.040735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:38.040681 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" Oct 31 00:36:38.912783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:38.912742 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" event={"ID":"9bda4c16-8ea5-4e7e-adda-8215696d6ce4","Type":"ContainerDied","Data":"43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c"} Oct 31 00:36:38.912783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:38.912772 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38" Oct 31 00:36:38.912783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:38.912787 5577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43f99d3213917ba7fab3cb301381a312cad84c3ff3a84d03996272e37f123a5c" Oct 31 00:36:39.644330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:39.644293 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38"] Oct 31 00:36:39.648743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:39.648696 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/thanos-ingress-prometheus-18e376291ae0ba38"] Oct 31 00:36:41.129276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:41.129247 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bda4c16-8ea5-4e7e-adda-8215696d6ce4" path="/var/lib/kubelet/pods/9bda4c16-8ea5-4e7e-adda-8215696d6ce4/volumes" Oct 31 00:36:52.181880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:52.181811 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:52.312764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:52.312728 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:56.070413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:36:56.070380 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:36:59.291743 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:36:59.291637 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:37:00.358020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:00.357987 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:01.013138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:01.013107 5577 generic.go:356] "Generic (PLEG): container finished" podID="c2d7942b-59c9-4d34-bf75-4156c16e5bbd" containerID="fc446824bbd388f4c8c612a5340001dcc3d5b6840796a2efc40f7973a0c77769" exitCode=137 Oct 31 00:37:01.013320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:01.013152 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerDied","Data":"fc446824bbd388f4c8c612a5340001dcc3d5b6840796a2efc40f7973a0c77769"} Oct 31 00:37:01.013320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:01.013198 5577 scope.go:122] "RemoveContainer" containerID="fefbab087abcbb36cce9c5ca2a51e8677c1552fbc5bd7daa1b06c50e74d894b5" Oct 31 00:37:01.013570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:01.013539 5577 scope.go:122] "RemoveContainer" containerID="fc446824bbd388f4c8c612a5340001dcc3d5b6840796a2efc40f7973a0c77769" Oct 31 00:37:01.160819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:01.160787 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:37:02.019192 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:02.019162 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" event={"ID":"c2d7942b-59c9-4d34-bf75-4156c16e5bbd","Type":"ContainerStarted","Data":"fc0fcf3232e98dc73ca2d0f205e0ee8785136cc28c68d7d34d759bf906baf314"} Oct 31 00:37:02.019597 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:02.019256 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:37:06.077482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:06.077445 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:06.204967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:06.204933 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:09.853286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:09.853244 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:09.985969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:09.985935 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:10.406900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:10.406865 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:10.407126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:10.407104 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="829c41c6-7add-4cd1-a56d-811b87b13e7e" containerName="otc-container" containerID="cri-o://56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a" gracePeriod=30 Oct 31 00:37:10.534650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:10.534610 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:10.773844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:10.773805 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:11.592112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.592076 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:11.592637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.592524 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9bda4c16-8ea5-4e7e-adda-8215696d6ce4" containerName="curl" Oct 31 00:37:11.592637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.592539 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bda4c16-8ea5-4e7e-adda-8215696d6ce4" containerName="curl" Oct 31 00:37:11.592637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.592608 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bda4c16-8ea5-4e7e-adda-8215696d6ce4" containerName="curl" Oct 31 00:37:11.596009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.595977 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.601235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.601187 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:37:11.601399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.601375 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:37:11.601796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.601779 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-xlpw8\"" Oct 31 00:37:11.601855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.601807 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:37:11.609728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.609688 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:11.669186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.669137 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:37:11.669186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.669137 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:37:11.669186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.669140 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:37:11.669186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.669177 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:37:11.675880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.675838 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:11.728765 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.728699 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.728952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.728816 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.728952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.728853 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.728952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.728881 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.728952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.728898 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.729116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.729055 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.729116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.729101 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.729219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.729121 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk7qk\" (UniqueName: \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.826028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.825991 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:11.830033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830012 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830122 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830050 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830122 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830067 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830122 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830104 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830127 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830144 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk7qk\" (UniqueName: \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830199 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830236 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830744 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830784 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.830959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830936 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.831029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.830960 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.833272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.833241 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.833272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.833255 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.833413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.833302 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:11.843461 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:11.843396 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk7qk\" (UniqueName: \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:12.745724 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.745678 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:12.840841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840764 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.840841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840812 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.840841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840835 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.841079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840901 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.841079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840936 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\" (UniqueName: \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.841079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.840971 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\") pod \"829c41c6-7add-4cd1-a56d-811b87b13e7e\" (UID: \"829c41c6-7add-4cd1-a56d-811b87b13e7e\") " Oct 31 00:37:12.841265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.841239 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:12.843210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.843179 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:12.843360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.843208 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:12.843360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.843327 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "kube-api-access-d854s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:12.843605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.843587 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:12.843672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.843600 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs" pod "829c41c6-7add-4cd1-a56d-811b87b13e7e" (UID: "829c41c6-7add-4cd1-a56d-811b87b13e7e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:12.942001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.941959 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/829c41c6-7add-4cd1-a56d-811b87b13e7e-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.942001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.941993 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.942001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.942003 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.942001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.942011 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.942274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.942020 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d854s\" (UniqueName: \"kubernetes.io/projected/829c41c6-7add-4cd1-a56d-811b87b13e7e-kube-api-access-d854s\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:12.942274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:12.942030 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/829c41c6-7add-4cd1-a56d-811b87b13e7e-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:13.025738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.025688 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-85cd6b576d-z5f58" Oct 31 00:37:13.075940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.075895 5577 generic.go:356] "Generic (PLEG): container finished" podID="829c41c6-7add-4cd1-a56d-811b87b13e7e" containerID="56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a" exitCode=0 Oct 31 00:37:13.076168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.075967 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"829c41c6-7add-4cd1-a56d-811b87b13e7e","Type":"ContainerDied","Data":"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a"} Oct 31 00:37:13.076168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.076013 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:13.076168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.076021 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"829c41c6-7add-4cd1-a56d-811b87b13e7e","Type":"ContainerDied","Data":"097552cdf878886409eff62dc65242d6f6c3ff587a9b89b09c2cdc6a8661eb64"} Oct 31 00:37:13.076168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.076052 5577 scope.go:122] "RemoveContainer" containerID="56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a" Oct 31 00:37:13.088403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.088376 5577 scope.go:122] "RemoveContainer" containerID="56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a" Oct 31 00:37:13.088788 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:13.088760 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a\": container with ID starting with 56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a not found: ID does not exist" containerID="56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a" Oct 31 00:37:13.088891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.088801 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a"} err="failed to get container status \"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a\": rpc error: code = NotFound desc = could not find container \"56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a\": container with ID starting with 56adaeec91e158256bc69613e04d6d797b5bcd8d93d9af8c5fe58acdcd0ef08a not found: ID does not exist" Oct 31 00:37:13.098256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.098195 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:13.102886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.102858 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:13.130508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:13.130479 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="829c41c6-7add-4cd1-a56d-811b87b13e7e" path="/var/lib/kubelet/pods/829c41c6-7add-4cd1-a56d-811b87b13e7e/volumes" Oct 31 00:37:18.592108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.592058 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.598564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.598536 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.692832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.692791 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.693005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.692945 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.696692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.696672 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.789745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.789684 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.807445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.807415 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:18.816880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:18.816855 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:19.106689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:19.106652 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:19.110997 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:37:19.110969 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2ba58b_f1e0_4dd1_872c_853641091c17.slice/crio-2c828dc7fa66a9302486b005f1f6bca9a57d1136e66b5d86e18202678e86acaa WatchSource:0}: Error finding container 2c828dc7fa66a9302486b005f1f6bca9a57d1136e66b5d86e18202678e86acaa: Status 404 returned error can't find the container with id 2c828dc7fa66a9302486b005f1f6bca9a57d1136e66b5d86e18202678e86acaa Oct 31 00:37:20.111814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.111778 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerStarted","Data":"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e"} Oct 31 00:37:20.112222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.111833 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerStarted","Data":"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379"} Oct 31 00:37:20.112222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.111848 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerStarted","Data":"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75"} Oct 31 00:37:20.112222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.111863 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerStarted","Data":"2c828dc7fa66a9302486b005f1f6bca9a57d1136e66b5d86e18202678e86acaa"} Oct 31 00:37:20.112222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.112016 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:20.114465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.114443 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:20.138672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:20.138615 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=9.138598125 podStartE2EDuration="9.138598125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:20.137072934 +0000 UTC m=+2279.128732551" watchObservedRunningTime="2026-10-31 00:37:20.138598125 +0000 UTC m=+2279.130257703" Oct 31 00:37:21.116505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:21.116474 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:21.116968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:21.116537 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:21.129175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:21.129149 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:22.117296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.117257 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.243323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.243287 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.702525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.702491 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.871093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.871059 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:22.881200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.881168 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:22.993281 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:22.993198 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:23.127086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:23.127046 5577 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" secret="" err="secret \"tempo-data-science-tempomonolithic-dockercfg-xlpw8\" not found" Oct 31 00:37:23.234114 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234080 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.234392 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234372 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.73435619 +0000 UTC m=+2282.726015756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.234392 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234106 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234424 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734414858 +0000 UTC m=+2282.726074454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234101 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234103 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234108 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234136 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234140 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:23.234611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234465 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734456238 +0000 UTC m=+2282.726115851 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:23.234918 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234640 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734620935 +0000 UTC m=+2282.726280498 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:23.234918 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234666 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734654996 +0000 UTC m=+2282.726314558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:23.234918 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234685 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734676143 +0000 UTC m=+2282.726335712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:23.234918 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.234722 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:23.734696484 +0000 UTC m=+2282.726356047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:23.739569 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739529 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.739569 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739546 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739602 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739589152 +0000 UTC m=+2283.731248740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739548 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739629 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739659 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739670 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739648985 +0000 UTC m=+2283.731308568 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739697 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739689879 +0000 UTC m=+2283.731349444 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739732 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739721419 +0000 UTC m=+2283.731380985 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:23.739835 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739748 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739741179 +0000 UTC m=+2283.731400744 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:23.740148 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739894 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:23.740148 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739904 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:23.740148 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739940 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739932558 +0000 UTC m=+2283.731592123 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:23.740148 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:23.739960 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:24.739944941 +0000 UTC m=+2283.731604523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:24.131204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:24.131160 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo" containerID="cri-o://5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75" gracePeriod=30 Oct 31 00:37:24.131204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:24.131187 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway-opa" containerID="cri-o://dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e" gracePeriod=30 Oct 31 00:37:24.131746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:24.131212 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway" containerID="cri-o://686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379" gracePeriod=30 Oct 31 00:37:24.749265 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749211 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:24.749265 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749251 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway: secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749280 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-config: configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749222 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-serving-cabundle: configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749311 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749290366 +0000 UTC m=+2285.740949941 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-tenants" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749229 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/tempo-data-science-tempomonolithic-ca-bundle: configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749328 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749319266 +0000 UTC m=+2285.740978833 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "gateway-rbac" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-gateway" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749344 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749336189 +0000 UTC m=+2285.740995754 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-conf" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-config" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749255 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-mtls: secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749358 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749349817 +0000 UTC m=+2285.741009383 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-serving-cabundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-serving-cabundle" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749251 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/tempo-data-science-tempomonolithic-gateway-serving-cert: secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749374 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749365627 +0000 UTC m=+2285.741025196 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : configmap "tempo-data-science-tempomonolithic-ca-bundle" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749388 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749380459 +0000 UTC m=+2285.741040028 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-mtls" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-mtls" not found Oct 31 00:37:24.749501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:24.749400 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert podName:9c2ba58b-f1e0-4dd1-872c-853641091c17 nodeName:}" failed. No retries permitted until 2026-10-31 00:37:26.749392989 +0000 UTC m=+2285.741052554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "tempo-data-science-tempomonolithic-gateway-serving-cert" (UniqueName: "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert") pod "tempo-data-science-tempomonolithic-0" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17") : secret "tempo-data-science-tempomonolithic-gateway-serving-cert" not found Oct 31 00:37:25.112313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.112274 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.133.0.93:8083/ready\": dial tcp 10.133.0.93:8083: connect: connection refused" Oct 31 00:37:25.142641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.142608 5577 generic.go:356] "Generic (PLEG): container finished" podID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerID="dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e" exitCode=0 Oct 31 00:37:25.142641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.142634 5577 generic.go:356] "Generic (PLEG): container finished" podID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerID="5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75" exitCode=0 Oct 31 00:37:25.143180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.142671 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerDied","Data":"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e"} Oct 31 00:37:25.143180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.142752 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerDied","Data":"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75"} Oct 31 00:37:25.387802 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.387776 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:25.454806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.454777 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.454982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.454829 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.454982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.454868 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.454982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.454888 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455070 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455121 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455158 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455224 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455230 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.455294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455239 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.455294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455256 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\" (UniqueName: \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\") pod \"9c2ba58b-f1e0-4dd1-872c-853641091c17\" (UID: \"9c2ba58b-f1e0-4dd1-872c-853641091c17\") " Oct 31 00:37:25.455294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455282 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.455566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455523 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.455566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455545 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-rbac\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.455566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455561 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.455802 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.455601 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:25.457379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.457351 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.457564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.457539 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.457835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.457816 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "kube-api-access-zk7qk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:25.457994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.457965 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:25.458607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.458586 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "9c2ba58b-f1e0-4dd1-872c-853641091c17" (UID: "9c2ba58b-f1e0-4dd1-872c-853641091c17"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:37:25.556321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556283 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-gateway-tenants\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.556321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556336 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:37:25.556559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556349 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.556559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556361 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.556559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556371 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c2ba58b-f1e0-4dd1-872c-853641091c17-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.556559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.556382 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk7qk\" (UniqueName: \"kubernetes.io/projected/9c2ba58b-f1e0-4dd1-872c-853641091c17-kube-api-access-zk7qk\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:25.565787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.565767 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:37:25.656829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:25.656758 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" Oct 31 00:37:26.148542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.148510 5577 generic.go:356] "Generic (PLEG): container finished" podID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerID="686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379" exitCode=0 Oct 31 00:37:26.148991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.148562 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:26.148991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.148591 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerDied","Data":"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379"} Oct 31 00:37:26.148991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.148645 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"9c2ba58b-f1e0-4dd1-872c-853641091c17","Type":"ContainerDied","Data":"2c828dc7fa66a9302486b005f1f6bca9a57d1136e66b5d86e18202678e86acaa"} Oct 31 00:37:26.148991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.148670 5577 scope.go:122] "RemoveContainer" containerID="dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e" Oct 31 00:37:26.157602 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.157589 5577 scope.go:122] "RemoveContainer" containerID="686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379" Oct 31 00:37:26.165177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.165159 5577 scope.go:122] "RemoveContainer" containerID="5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75" Oct 31 00:37:26.171218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.171196 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:26.174024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174005 5577 scope.go:122] "RemoveContainer" containerID="dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e" Oct 31 00:37:26.174282 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:26.174266 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e\": container with ID starting with dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e not found: ID does not exist" containerID="dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e" Oct 31 00:37:26.174349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174289 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e"} err="failed to get container status \"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e\": rpc error: code = NotFound desc = could not find container \"dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e\": container with ID starting with dca70826a134af56358bf406dc5b0ab91c270898eefe3150baac2fc0009cee0e not found: ID does not exist" Oct 31 00:37:26.174349 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174305 5577 scope.go:122] "RemoveContainer" containerID="686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379" Oct 31 00:37:26.174550 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:26.174533 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379\": container with ID starting with 686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379 not found: ID does not exist" containerID="686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379" Oct 31 00:37:26.174596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174557 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379"} err="failed to get container status \"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379\": rpc error: code = NotFound desc = could not find container \"686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379\": container with ID starting with 686e8a32efefc84a7d8b00136f3e1470a94b723a576387d1ecd12a3459138379 not found: ID does not exist" Oct 31 00:37:26.174596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174570 5577 scope.go:122] "RemoveContainer" containerID="5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75" Oct 31 00:37:26.174875 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:26.174849 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75\": container with ID starting with 5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75 not found: ID does not exist" containerID="5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75" Oct 31 00:37:26.174970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.174881 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75"} err="failed to get container status \"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75\": rpc error: code = NotFound desc = could not find container \"5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75\": container with ID starting with 5095f82afce47eb4f940589fc07c12171abe196e1dd8d4be7fcbc4e035b7aa75 not found: ID does not exist" Oct 31 00:37:26.176688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.176666 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:26.541225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:26.541187 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:27.129230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:27.129181 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" path="/var/lib/kubelet/pods/9c2ba58b-f1e0-4dd1-872c-853641091c17/volumes" Oct 31 00:37:29.293616 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:29.293582 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:37:34.718774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:34.718741 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:34.892987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:34.892944 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:35.391047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391005 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:35.391472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391451 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway-opa" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391475 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway-opa" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391501 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391508 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391526 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391534 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391550 5577 cpu_manager.go:414] "containerMap: removing container" podUID="829c41c6-7add-4cd1-a56d-811b87b13e7e" containerName="otc-container" Oct 31 00:37:35.391579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391556 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="829c41c6-7add-4cd1-a56d-811b87b13e7e" containerName="otc-container" Oct 31 00:37:35.391970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391645 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway" Oct 31 00:37:35.391970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391661 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="829c41c6-7add-4cd1-a56d-811b87b13e7e" containerName="otc-container" Oct 31 00:37:35.391970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391675 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo-gateway-opa" Oct 31 00:37:35.391970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.391687 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2ba58b-f1e0-4dd1-872c-853641091c17" containerName="tempo" Oct 31 00:37:35.396487 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.396467 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.403470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.403454 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-server-cert\"" Oct 31 00:37:35.404841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.404819 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator-dockercfg-gpb29\"" Oct 31 00:37:35.405747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.405728 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-targetallocator\"" Oct 31 00:37:35.406078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.406057 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:35.413722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.413680 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:35.538871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.538833 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxtg9\" (UniqueName: \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.539073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.538878 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.539073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.538904 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.543878 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.543829 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:35.639513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.639480 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.639513 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.639517 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.639734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.639575 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxtg9\" (UniqueName: \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.640138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.640119 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.642285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.642235 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.651754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.651704 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxtg9\" (UniqueName: \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\") pod \"data-science-collector-targetallocator-79895649b7-xxfj4\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.708286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.708256 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:35.982228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.982156 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:35.985557 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:37:35.985510 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28d6b776_31c4_4714_bf3f_91e0f5de7afc.slice/crio-549b790564ee3a3d509fcdf572f3abdddc49e1cf83dd27098831d8d7009aa466 WatchSource:0}: Error finding container 549b790564ee3a3d509fcdf572f3abdddc49e1cf83dd27098831d8d7009aa466: Status 404 returned error can't find the container with id 549b790564ee3a3d509fcdf572f3abdddc49e1cf83dd27098831d8d7009aa466 Oct 31 00:37:35.987935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.987903 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:35.988081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:35.987957 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:36.181537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.181501 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:36.185909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.185873 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.192628 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.192603 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:37:36.192839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.192649 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 31 00:37:36.193157 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193135 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 31 00:37:36.193290 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193141 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 31 00:37:36.193397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193176 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 31 00:37:36.193497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193231 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 31 00:37:36.193747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193700 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-6dqqv\"" Oct 31 00:37:36.193890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.193872 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 31 00:37:36.196735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.195298 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" event={"ID":"28d6b776-31c4-4714-bf3f-91e0f5de7afc","Type":"ContainerStarted","Data":"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e"} Oct 31 00:37:36.196735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.195341 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" event={"ID":"28d6b776-31c4-4714-bf3f-91e0f5de7afc","Type":"ContainerStarted","Data":"549b790564ee3a3d509fcdf572f3abdddc49e1cf83dd27098831d8d7009aa466"} Oct 31 00:37:36.196735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.196201 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:36.198168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.198148 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:36.201140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.201107 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:36.243459 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.243378 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" podStartSLOduration=1.243366298 podStartE2EDuration="1.243366298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:36.242880025 +0000 UTC m=+2295.234539609" watchObservedRunningTime="2026-10-31 00:37:36.243366298 +0000 UTC m=+2295.235025883" Oct 31 00:37:36.284739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.284674 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:36.346944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.346907 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.346944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.346944 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347038 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347075 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347094 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347275 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxjwc\" (UniqueName: \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347481 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.347557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.347510 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.414174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.414128 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:36.448831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448799 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448833 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448870 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448894 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448931 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448961 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449003 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.448988 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.449019 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxjwc\" (UniqueName: \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.449772 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.449906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.449779 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.450058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.450035 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.450265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.450237 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.451942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.451919 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.452263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.452244 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.452618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.452597 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:36.462014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:36.461990 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxjwc\" (UniqueName: \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:38.870348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:38.870312 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:38.877653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:38.877627 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:38.971306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:38.971271 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:38.971476 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:38.971447 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:38.975080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:38.975064 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:39.065757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:39.065722 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/151619720cb39d40781bf258206fb0be746b77e5f55879e189e4cf0d6e46dfb0/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:39.082465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:39.082437 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:39.208484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:39.208404 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:39.488301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:39.488275 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:39.490545 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:37:39.490512 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d5dac6f_ebe6_4ac2_8f82_a9c59752558b.slice/crio-e2a106d5365730dc19b2350d3b8ac0f1a58318d6491a1b7573829ae96ba0eb64 WatchSource:0}: Error finding container e2a106d5365730dc19b2350d3b8ac0f1a58318d6491a1b7573829ae96ba0eb64: Status 404 returned error can't find the container with id e2a106d5365730dc19b2350d3b8ac0f1a58318d6491a1b7573829ae96ba0eb64 Oct 31 00:37:40.216850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.216812 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerStarted","Data":"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da"} Oct 31 00:37:40.217352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.216870 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerStarted","Data":"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe"} Oct 31 00:37:40.217352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.216891 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerStarted","Data":"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60"} Oct 31 00:37:40.217352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.216911 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerStarted","Data":"e2a106d5365730dc19b2350d3b8ac0f1a58318d6491a1b7573829ae96ba0eb64"} Oct 31 00:37:40.217352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.216941 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:40.219691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.219667 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:40.243147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:40.243108 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.243094821 podStartE2EDuration="4.243094821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:37:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:40.242885891 +0000 UTC m=+2299.234545514" watchObservedRunningTime="2026-10-31 00:37:40.243094821 +0000 UTC m=+2299.234754402" Oct 31 00:37:41.224447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:41.224411 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:41.225024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:41.224478 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:41.234121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:41.234090 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:46.924197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:46.924149 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.073361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.073329 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.555316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.555279 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.683521 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.683485 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:47.749632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.749586 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:47.749929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.749892 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo" containerID="cri-o://d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60" gracePeriod=30 Oct 31 00:37:47.750080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.750036 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway-opa" containerID="cri-o://c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da" gracePeriod=30 Oct 31 00:37:47.750080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.750064 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway" containerID="cri-o://a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe" gracePeriod=30 Oct 31 00:37:47.834848 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:47.834758 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:48.256583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.256553 5577 generic.go:356] "Generic (PLEG): container finished" podID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerID="c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da" exitCode=0 Oct 31 00:37:48.256583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.256579 5577 generic.go:356] "Generic (PLEG): container finished" podID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerID="d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60" exitCode=0 Oct 31 00:37:48.257066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.256634 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerDied","Data":"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da"} Oct 31 00:37:48.257066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.256703 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerDied","Data":"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60"} Oct 31 00:37:48.856489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.856455 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:48.856684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:48.856648 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" podUID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" containerName="ta-container" containerID="cri-o://61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e" gracePeriod=30 Oct 31 00:37:49.008262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.008232 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:49.120440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.120418 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:49.161446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161415 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161475 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161519 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\" (UniqueName: \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161608 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161548 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161807 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161672 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161807 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161776 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161822 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161902 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.161964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161912 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.161964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161917 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.161964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.161946 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\") pod \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\" (UID: \"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b\") " Oct 31 00:37:49.162413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.162192 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.162413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.162215 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-conf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.162413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.162270 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.162413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.162306 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.163996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.163971 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "kube-api-access-dxjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:49.164220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.164188 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.164569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.164550 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "pvc-4491ac70-4597-43a7-a792-f0813314e433". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:37:49.164679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.164648 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.164760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.164671 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants" pod "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" (UID: "8d5dac6f-ebe6-4ac2-8f82-a9c59752558b"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.262298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262262 5577 generic.go:356] "Generic (PLEG): container finished" podID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" containerID="61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e" exitCode=0 Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262303 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262303 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" event={"ID":"28d6b776-31c4-4714-bf3f-91e0f5de7afc","Type":"ContainerDied","Data":"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e"} Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262353 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4" event={"ID":"28d6b776-31c4-4714-bf3f-91e0f5de7afc","Type":"ContainerDied","Data":"549b790564ee3a3d509fcdf572f3abdddc49e1cf83dd27098831d8d7009aa466"} Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262372 5577 scope.go:122] "RemoveContainer" containerID="61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e" Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262439 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\") pod \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262514 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\" (UniqueName: \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\") pod \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " Oct 31 00:37:49.262695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262572 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\" (UniqueName: \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\") pod \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\" (UID: \"28d6b776-31c4-4714-bf3f-91e0f5de7afc\") " Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262781 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262815 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxjwc\" (UniqueName: \"kubernetes.io/projected/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-kube-api-access-dxjwc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262833 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262849 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-rbac\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262864 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262880 5577 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263106 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.262893 5577 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b-gateway-tenants\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.263439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.263249 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal" pod "28d6b776-31c4-4714-bf3f-91e0f5de7afc" (UID: "28d6b776-31c4-4714-bf3f-91e0f5de7afc"). InnerVolumeSpecName "ta-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:49.265105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.265079 5577 generic.go:356] "Generic (PLEG): container finished" podID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerID="a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe" exitCode=0 Oct 31 00:37:49.265226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.265114 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerDied","Data":"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe"} Oct 31 00:37:49.265226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.265139 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 31 00:37:49.265226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.265155 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8d5dac6f-ebe6-4ac2-8f82-a9c59752558b","Type":"ContainerDied","Data":"e2a106d5365730dc19b2350d3b8ac0f1a58318d6491a1b7573829ae96ba0eb64"} Oct 31 00:37:49.272614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.272592 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert" pod "28d6b776-31c4-4714-bf3f-91e0f5de7afc" (UID: "28d6b776-31c4-4714-bf3f-91e0f5de7afc"). InnerVolumeSpecName "data-science-collector-ta-server-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:49.272614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.272597 5577 scope.go:122] "RemoveContainer" containerID="61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e" Oct 31 00:37:49.272850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.272747 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9" pod "28d6b776-31c4-4714-bf3f-91e0f5de7afc" (UID: "28d6b776-31c4-4714-bf3f-91e0f5de7afc"). InnerVolumeSpecName "kube-api-access-fxtg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:49.272993 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:49.272960 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e\": container with ID starting with 61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e not found: ID does not exist" containerID="61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e" Oct 31 00:37:49.273040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.273002 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e"} err="failed to get container status \"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e\": rpc error: code = NotFound desc = could not find container \"61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e\": container with ID starting with 61dad319f7d4597d884fe3a43590c9467b7c72cc670c1242aeeecccb196bf79e not found: ID does not exist" Oct 31 00:37:49.273040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.273025 5577 scope.go:122] "RemoveContainer" containerID="c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da" Oct 31 00:37:49.275819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.275801 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-4491ac70-4597-43a7-a792-f0813314e433" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:37:49.281907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.281889 5577 scope.go:122] "RemoveContainer" containerID="a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe" Oct 31 00:37:49.289104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.289080 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:49.290061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.290041 5577 scope.go:122] "RemoveContainer" containerID="d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60" Oct 31 00:37:49.295469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.295447 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 31 00:37:49.299880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.299820 5577 scope.go:122] "RemoveContainer" containerID="c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da" Oct 31 00:37:49.300319 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:49.300297 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da\": container with ID starting with c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da not found: ID does not exist" containerID="c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da" Oct 31 00:37:49.300416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.300324 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da"} err="failed to get container status \"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da\": rpc error: code = NotFound desc = could not find container \"c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da\": container with ID starting with c6878654acca673bc8a5abefa0a876fbe412f3e828f5b820fd59c67a455994da not found: ID does not exist" Oct 31 00:37:49.300416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.300340 5577 scope.go:122] "RemoveContainer" containerID="a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe" Oct 31 00:37:49.300625 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:49.300607 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe\": container with ID starting with a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe not found: ID does not exist" containerID="a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe" Oct 31 00:37:49.300676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.300629 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe"} err="failed to get container status \"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe\": rpc error: code = NotFound desc = could not find container \"a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe\": container with ID starting with a9fbc97adaf6df68708193635ae25d191c017b8a59702918a7fc271c90f6b8fe not found: ID does not exist" Oct 31 00:37:49.300676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.300644 5577 scope.go:122] "RemoveContainer" containerID="d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60" Oct 31 00:37:49.300926 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:49.300911 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60\": container with ID starting with d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60 not found: ID does not exist" containerID="d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60" Oct 31 00:37:49.300980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.300929 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60"} err="failed to get container status \"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60\": rpc error: code = NotFound desc = could not find container \"d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60\": container with ID starting with d2947f3810c2784c650c9b09e2d154c31be2f50ca3ea7277fcf7ca179c645e60 not found: ID does not exist" Oct 31 00:37:49.363586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.363511 5577 reconciler_common.go:299] "Volume detached for volume \"ta-internal\" (UniqueName: \"kubernetes.io/configmap/28d6b776-31c4-4714-bf3f-91e0f5de7afc-ta-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.363586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.363540 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-4491ac70-4597-43a7-a792-f0813314e433\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0448bf0057f34f062\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-c917ea34eee3da550694297af3853ca45b80de2d5b97d30fcbb3560006edee83\"" Oct 31 00:37:49.363586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.363552 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-server-cert\" (UniqueName: \"kubernetes.io/secret/28d6b776-31c4-4714-bf3f-91e0f5de7afc-data-science-collector-ta-server-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.363586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.363562 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxtg9\" (UniqueName: \"kubernetes.io/projected/28d6b776-31c4-4714-bf3f-91e0f5de7afc-kube-api-access-fxtg9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:49.582994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.582961 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:49.586978 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:49.586956 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-targetallocator-79895649b7-xxfj4"] Oct 31 00:37:50.143450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143379 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:50.143734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143722 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo" Oct 31 00:37:50.143791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143736 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo" Oct 31 00:37:50.143791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143759 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway-opa" Oct 31 00:37:50.143791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143764 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway-opa" Oct 31 00:37:50.143791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143774 5577 cpu_manager.go:414] "containerMap: removing container" podUID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" containerName="ta-container" Oct 31 00:37:50.143791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143779 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" containerName="ta-container" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143797 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143802 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143848 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway-opa" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143856 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" containerName="ta-container" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143865 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo-gateway" Oct 31 00:37:50.143933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.143871 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" containerName="tempo" Oct 31 00:37:50.148467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.148450 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.153030 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.153010 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cfb2n\"" Oct 31 00:37:50.153135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.153059 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:37:50.153135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.153101 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:37:50.154903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.154838 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:37:50.158210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.158187 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:37:50.159497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.159473 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:50.252964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.252935 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:37:50.270973 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.270937 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.270973 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.270976 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfbm8\" (UniqueName: \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.271482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.271092 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.271482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.271136 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.271482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.271223 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.271482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.271312 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372376 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372421 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfbm8\" (UniqueName: \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372466 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372496 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372529 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.372846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.372734 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.373626 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.373593 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.375236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.375204 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.375236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.375233 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.375414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.375392 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.375474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.375418 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.385009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.384986 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfbm8\" (UniqueName: \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\") pod \"data-science-collector-collector-1\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.461058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.460981 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:50.737225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.737139 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:50.739795 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:37:50.739768 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1406343_3f9e_42b7_a136_a18849175421.slice/crio-a531afecb63375ef6bb572928970f7640a75e36b0b4553646d227231ae8c83ac WatchSource:0}: Error finding container a531afecb63375ef6bb572928970f7640a75e36b0b4553646d227231ae8c83ac: Status 404 returned error can't find the container with id a531afecb63375ef6bb572928970f7640a75e36b0b4553646d227231ae8c83ac Oct 31 00:37:50.742096 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.742065 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:50.742201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.742135 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:37:50.994120 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:50.994025 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.122679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.122639 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:51.129697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.129669 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28d6b776-31c4-4714-bf3f-91e0f5de7afc" path="/var/lib/kubelet/pods/28d6b776-31c4-4714-bf3f-91e0f5de7afc/volumes" Oct 31 00:37:51.130112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.130099 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d5dac6f-ebe6-4ac2-8f82-a9c59752558b" path="/var/lib/kubelet/pods/8d5dac6f-ebe6-4ac2-8f82-a9c59752558b/volumes" Oct 31 00:37:51.277225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.277191 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e1406343-3f9e-42b7-a136-a18849175421","Type":"ContainerStarted","Data":"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921"} Oct 31 00:37:51.277735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.277242 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e1406343-3f9e-42b7-a136-a18849175421","Type":"ContainerStarted","Data":"a531afecb63375ef6bb572928970f7640a75e36b0b4553646d227231ae8c83ac"} Oct 31 00:37:51.298841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:51.298791 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.298773374 podStartE2EDuration="1.298773374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:51.297881027 +0000 UTC m=+2310.289540612" watchObservedRunningTime="2026-10-31 00:37:51.298773374 +0000 UTC m=+2310.290432958" Oct 31 00:37:53.222295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.222259 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:37:53.222700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.222490 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="kube-rbac-proxy" containerID="cri-o://5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" gracePeriod=30 Oct 31 00:37:53.222700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.222580 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="prom-label-proxy" containerID="cri-o://f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" gracePeriod=30 Oct 31 00:37:53.222700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.222589 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="method-gate" containerID="cri-o://5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" gracePeriod=30 Oct 31 00:37:53.225340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.225301 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:37:53.225528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.225510 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="kube-rbac-proxy" containerID="cri-o://2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" gracePeriod=30 Oct 31 00:37:53.225638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.225618 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="prom-label-proxy" containerID="cri-o://b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" gracePeriod=30 Oct 31 00:37:53.225696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.225677 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="method-gate" containerID="cri-o://d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" gracePeriod=30 Oct 31 00:37:53.426965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.426919 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.429614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.429226 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:37:53.429614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.429467 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="alertmanager" containerID="cri-o://191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" gracePeriod=120 Oct 31 00:37:53.429614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.429503 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="config-reloader" containerID="cri-o://50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" gracePeriod=120 Oct 31 00:37:53.511568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.511537 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:37:53.541849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.541821 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:37:53.554988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.554955 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.590449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.590416 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:53.590653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.590629 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="e1406343-3f9e-42b7-a136-a18849175421" containerName="otc-container" containerID="cri-o://15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921" gracePeriod=30 Oct 31 00:37:53.598286 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598252 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\") pod \"a7f8fa61-d4dd-4a65-9015-249bd2067703\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " Oct 31 00:37:53.598439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598323 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\" (UniqueName: \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\") pod \"a7f8fa61-d4dd-4a65-9015-249bd2067703\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " Oct 31 00:37:53.598439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598361 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\" (UniqueName: \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\") pod \"a7f8fa61-d4dd-4a65-9015-249bd2067703\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " Oct 31 00:37:53.598439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598419 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\") pod \"a7f8fa61-d4dd-4a65-9015-249bd2067703\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " Oct 31 00:37:53.598554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598499 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\") pod \"a7f8fa61-d4dd-4a65-9015-249bd2067703\" (UID: \"a7f8fa61-d4dd-4a65-9015-249bd2067703\") " Oct 31 00:37:53.599007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.598978 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config" pod "a7f8fa61-d4dd-4a65-9015-249bd2067703" (UID: "a7f8fa61-d4dd-4a65-9015-249bd2067703"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.599196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.599174 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca" pod "a7f8fa61-d4dd-4a65-9015-249bd2067703" (UID: "a7f8fa61-d4dd-4a65-9015-249bd2067703"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.599272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.599203 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca" pod "a7f8fa61-d4dd-4a65-9015-249bd2067703" (UID: "a7f8fa61-d4dd-4a65-9015-249bd2067703"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.665366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.665325 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls" pod "a7f8fa61-d4dd-4a65-9015-249bd2067703" (UID: "a7f8fa61-d4dd-4a65-9015-249bd2067703"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.665366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.665350 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn" pod "a7f8fa61-d4dd-4a65-9015-249bd2067703" (UID: "a7f8fa61-d4dd-4a65-9015-249bd2067703"). InnerVolumeSpecName "kube-api-access-kr7dn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.670499 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.670468 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:37:53.670769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.670729 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" containerName="perses" containerID="cri-o://5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf" gracePeriod=30 Oct 31 00:37:53.687176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.687129 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.699432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699397 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\" (UniqueName: \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\") pod \"e99107d2-526a-494e-accd-e32ca14a5507\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " Oct 31 00:37:53.699620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699467 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\" (UniqueName: \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\") pod \"e99107d2-526a-494e-accd-e32ca14a5507\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " Oct 31 00:37:53.699620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699517 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\" (UniqueName: \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\") pod \"e99107d2-526a-494e-accd-e32ca14a5507\" (UID: \"e99107d2-526a-494e-accd-e32ca14a5507\") " Oct 31 00:37:53.699801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699775 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7dn\" (UniqueName: \"kubernetes.io/projected/a7f8fa61-d4dd-4a65-9015-249bd2067703-kube-api-access-kr7dn\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.699801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699794 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a7f8fa61-d4dd-4a65-9015-249bd2067703-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.699916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699810 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.699916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699825 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.699916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.699839 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a7f8fa61-d4dd-4a65-9015-249bd2067703-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.704741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.700190 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config" pod "e99107d2-526a-494e-accd-e32ca14a5507" (UID: "e99107d2-526a-494e-accd-e32ca14a5507"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:53.704741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.702553 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls" pod "e99107d2-526a-494e-accd-e32ca14a5507" (UID: "e99107d2-526a-494e-accd-e32ca14a5507"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:53.704741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.703556 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc" pod "e99107d2-526a-494e-accd-e32ca14a5507" (UID: "e99107d2-526a-494e-accd-e32ca14a5507"). InnerVolumeSpecName "kube-api-access-2jjtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:53.801634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.801590 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e99107d2-526a-494e-accd-e32ca14a5507-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.801634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.801633 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e99107d2-526a-494e-accd-e32ca14a5507-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.801899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.801648 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jjtc\" (UniqueName: \"kubernetes.io/projected/e99107d2-526a-494e-accd-e32ca14a5507-kube-api-access-2jjtc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:53.827013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.821561 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.956925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.956894 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:53.961486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.961465 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:53.991189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:53.991164 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:37:54.088828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.088783 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:54.104098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104073 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\" (UniqueName: \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\") pod \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " Oct 31 00:37:54.104206 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104114 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\" (UniqueName: \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.104206 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104146 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\" (UniqueName: \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\") pod \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " Oct 31 00:37:54.104305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104233 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " Oct 31 00:37:54.104305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104268 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.104305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104304 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\" (UniqueName: \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.104473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104321 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.104473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104367 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.104473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104397 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\" (UniqueName: \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\") pod \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\" (UID: \"be23abc0-0155-4ed5-8c30-fd61fa5bde5e\") " Oct 31 00:37:54.104473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.104413 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\") pod \"e1406343-3f9e-42b7-a136-a18849175421\" (UID: \"e1406343-3f9e-42b7-a136-a18849175421\") " Oct 31 00:37:54.106054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.106007 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.106054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.106022 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config" pod "be23abc0-0155-4ed5-8c30-fd61fa5bde5e" (UID: "be23abc0-0155-4ed5-8c30-fd61fa5bde5e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:37:54.109609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.109581 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "kube-api-access-lfbm8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.109609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.109645 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf" pod "be23abc0-0155-4ed5-8c30-fd61fa5bde5e" (UID: "be23abc0-0155-4ed5-8c30-fd61fa5bde5e"). InnerVolumeSpecName "kube-api-access-97dbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.109865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.109828 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "be23abc0-0155-4ed5-8c30-fd61fa5bde5e" (UID: "be23abc0-0155-4ed5-8c30-fd61fa5bde5e"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:37:54.110979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.110958 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.111042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.110995 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.112283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.112256 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.114798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.114776 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert" pod "e1406343-3f9e-42b7-a136-a18849175421" (UID: "e1406343-3f9e-42b7-a136-a18849175421"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.187554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.187524 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:37:54.206114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206076 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206113 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/e1406343-3f9e-42b7-a136-a18849175421-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206129 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206144 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206159 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206173 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1406343-3f9e-42b7-a136-a18849175421-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206186 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97dbf\" (UniqueName: \"kubernetes.io/projected/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-kube-api-access-97dbf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206200 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfbm8\" (UniqueName: \"kubernetes.io/projected/e1406343-3f9e-42b7-a136-a18849175421-kube-api-access-lfbm8\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.206386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.206233 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:37:54.225438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.225412 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:37:54.268607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.268572 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins" pod "be23abc0-0155-4ed5-8c30-fd61fa5bde5e" (UID: "be23abc0-0155-4ed5-8c30-fd61fa5bde5e"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:37:54.293270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293230 5577 generic.go:356] "Generic (PLEG): container finished" podID="e99107d2-526a-494e-accd-e32ca14a5507" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" exitCode=0 Oct 31 00:37:54.293270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293260 5577 generic.go:356] "Generic (PLEG): container finished" podID="e99107d2-526a-494e-accd-e32ca14a5507" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" exitCode=0 Oct 31 00:37:54.293270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293268 5577 generic.go:356] "Generic (PLEG): container finished" podID="e99107d2-526a-494e-accd-e32ca14a5507" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" exitCode=0 Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293293 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293312 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerDied","Data":"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841"} Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293387 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerDied","Data":"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d"} Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293435 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerDied","Data":"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3"} Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293459 5577 scope.go:122] "RemoveContainer" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" Oct 31 00:37:54.293539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.293465 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7" event={"ID":"e99107d2-526a-494e-accd-e32ca14a5507","Type":"ContainerDied","Data":"45b1fc06aed0a763338d2313158a0b1cefbb63d8821086877bd4eea41f196316"} Oct 31 00:37:54.295528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295505 5577 generic.go:356] "Generic (PLEG): container finished" podID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" exitCode=0 Oct 31 00:37:54.295528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295532 5577 generic.go:356] "Generic (PLEG): container finished" podID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" exitCode=0 Oct 31 00:37:54.295528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295543 5577 generic.go:356] "Generic (PLEG): container finished" podID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" exitCode=0 Oct 31 00:37:54.295819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295630 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" Oct 31 00:37:54.295819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295628 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerDied","Data":"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8"} Oct 31 00:37:54.295819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295780 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerDied","Data":"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921"} Oct 31 00:37:54.295979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295824 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerDied","Data":"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72"} Oct 31 00:37:54.295979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.295859 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6" event={"ID":"a7f8fa61-d4dd-4a65-9015-249bd2067703","Type":"ContainerDied","Data":"e1d4f0054cbe296c05ce1b770a93bf21420854ba8bb64714cc144df6e93fbc45"} Oct 31 00:37:54.297500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.297159 5577 generic.go:356] "Generic (PLEG): container finished" podID="e1406343-3f9e-42b7-a136-a18849175421" containerID="15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921" exitCode=0 Oct 31 00:37:54.297500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.297217 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:37:54.297500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.297236 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e1406343-3f9e-42b7-a136-a18849175421","Type":"ContainerDied","Data":"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921"} Oct 31 00:37:54.297500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.297282 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"e1406343-3f9e-42b7-a136-a18849175421","Type":"ContainerDied","Data":"a531afecb63375ef6bb572928970f7640a75e36b0b4553646d227231ae8c83ac"} Oct 31 00:37:54.299866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299816 5577 generic.go:356] "Generic (PLEG): container finished" podID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerID="50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" exitCode=0 Oct 31 00:37:54.299866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299843 5577 generic.go:356] "Generic (PLEG): container finished" podID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerID="191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" exitCode=0 Oct 31 00:37:54.300061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299881 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:37:54.300061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299895 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerDied","Data":"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf"} Oct 31 00:37:54.300061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299950 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerDied","Data":"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1"} Oct 31 00:37:54.300061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.299983 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"4e97df37-be64-4b72-a93b-9dbf940ca86b","Type":"ContainerDied","Data":"3e561738106b7e11df3376e6c8b2cc840fe17799bec235238468526bc381ba52"} Oct 31 00:37:54.301428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.301404 5577 generic.go:356] "Generic (PLEG): container finished" podID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" containerID="5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf" exitCode=0 Oct 31 00:37:54.301543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.301456 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:37:54.301543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.301463 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"be23abc0-0155-4ed5-8c30-fd61fa5bde5e","Type":"ContainerDied","Data":"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf"} Oct 31 00:37:54.301543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.301491 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"be23abc0-0155-4ed5-8c30-fd61fa5bde5e","Type":"ContainerDied","Data":"86ac9315112a1cd51390b642d2d378a38be5da9639d864b4d02a929605439030"} Oct 31 00:37:54.306344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.306321 5577 scope.go:122] "RemoveContainer" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" Oct 31 00:37:54.311517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.311484 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.311668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.311579 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.311668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.311646 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.311851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.311685 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.312600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.312443 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.312809 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.312763 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:37:54.313140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.313119 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.313236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.313171 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\") pod \"4e97df37-be64-4b72-a93b-9dbf940ca86b\" (UID: \"4e97df37-be64-4b72-a93b-9dbf940ca86b\") " Oct 31 00:37:54.315037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.315011 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.315419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.315393 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "kube-api-access-v9rph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:37:54.316055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.315785 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/be23abc0-0155-4ed5-8c30-fd61fa5bde5e-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.318780 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:37:54.318924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.318808 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.318826 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.318841 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rph\" (UniqueName: \"kubernetes.io/projected/4e97df37-be64-4b72-a93b-9dbf940ca86b-kube-api-access-v9rph\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.318924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.316135 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.318924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.317001 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:37:54.322340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.322318 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.328250 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.328224 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config" pod "4e97df37-be64-4b72-a93b-9dbf940ca86b" (UID: "4e97df37-be64-4b72-a93b-9dbf940ca86b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:37:54.339269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.339246 5577 scope.go:122] "RemoveContainer" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" Oct 31 00:37:54.347248 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.347223 5577 scope.go:122] "RemoveContainer" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" Oct 31 00:37:54.347627 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.347596 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": container with ID starting with b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841 not found: ID does not exist" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" Oct 31 00:37:54.347895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.347639 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841"} err="failed to get container status \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": rpc error: code = NotFound desc = could not find container \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": container with ID starting with b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841 not found: ID does not exist" Oct 31 00:37:54.347895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.347662 5577 scope.go:122] "RemoveContainer" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" Oct 31 00:37:54.348019 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.347991 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": container with ID starting with d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d not found: ID does not exist" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" Oct 31 00:37:54.348069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348028 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d"} err="failed to get container status \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": rpc error: code = NotFound desc = could not find container \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": container with ID starting with d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d not found: ID does not exist" Oct 31 00:37:54.348069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348052 5577 scope.go:122] "RemoveContainer" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" Oct 31 00:37:54.348321 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.348303 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": container with ID starting with 2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3 not found: ID does not exist" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" Oct 31 00:37:54.348354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348328 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3"} err="failed to get container status \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": rpc error: code = NotFound desc = could not find container \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": container with ID starting with 2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3 not found: ID does not exist" Oct 31 00:37:54.348354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348343 5577 scope.go:122] "RemoveContainer" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" Oct 31 00:37:54.348581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348561 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841"} err="failed to get container status \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": rpc error: code = NotFound desc = could not find container \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": container with ID starting with b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841 not found: ID does not exist" Oct 31 00:37:54.348625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348582 5577 scope.go:122] "RemoveContainer" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" Oct 31 00:37:54.348625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348602 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:37:54.348840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348821 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d"} err="failed to get container status \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": rpc error: code = NotFound desc = could not find container \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": container with ID starting with d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d not found: ID does not exist" Oct 31 00:37:54.348906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.348843 5577 scope.go:122] "RemoveContainer" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" Oct 31 00:37:54.349091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349073 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3"} err="failed to get container status \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": rpc error: code = NotFound desc = could not find container \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": container with ID starting with 2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3 not found: ID does not exist" Oct 31 00:37:54.349136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349092 5577 scope.go:122] "RemoveContainer" containerID="b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841" Oct 31 00:37:54.349330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349311 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841"} err="failed to get container status \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": rpc error: code = NotFound desc = could not find container \"b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841\": container with ID starting with b4189b1df75edd9afcf3011c96c4f9d40a90556a1590e5a72ef02536aacf8841 not found: ID does not exist" Oct 31 00:37:54.349401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349331 5577 scope.go:122] "RemoveContainer" containerID="d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d" Oct 31 00:37:54.349547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349530 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d"} err="failed to get container status \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": rpc error: code = NotFound desc = could not find container \"d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d\": container with ID starting with d737331bf1b387d3f7b5f3c7ca271ac7ad98c7deeefb86dee1babc6b3611ff7d not found: ID does not exist" Oct 31 00:37:54.349596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349548 5577 scope.go:122] "RemoveContainer" containerID="2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3" Oct 31 00:37:54.349805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349787 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3"} err="failed to get container status \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": rpc error: code = NotFound desc = could not find container \"2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3\": container with ID starting with 2be834f4786e3b0a6a4058eb9f13aa03047650dfed5586cb08381b03e7bb25a3 not found: ID does not exist" Oct 31 00:37:54.349870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.349806 5577 scope.go:122] "RemoveContainer" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" Oct 31 00:37:54.353281 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.353261 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bzrr7"] Oct 31 00:37:54.360532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.360493 5577 scope.go:122] "RemoveContainer" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" Oct 31 00:37:54.365649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.365617 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:37:54.368896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.368876 5577 scope.go:122] "RemoveContainer" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" Oct 31 00:37:54.371137 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.371118 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-zw2h6"] Oct 31 00:37:54.377232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.377213 5577 scope.go:122] "RemoveContainer" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" Oct 31 00:37:54.377499 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.377481 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": container with ID starting with f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8 not found: ID does not exist" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" Oct 31 00:37:54.377554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.377507 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8"} err="failed to get container status \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": rpc error: code = NotFound desc = could not find container \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": container with ID starting with f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8 not found: ID does not exist" Oct 31 00:37:54.377554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.377523 5577 scope.go:122] "RemoveContainer" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" Oct 31 00:37:54.377833 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.377813 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": container with ID starting with 5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921 not found: ID does not exist" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" Oct 31 00:37:54.377935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.377839 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921"} err="failed to get container status \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": rpc error: code = NotFound desc = could not find container \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": container with ID starting with 5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921 not found: ID does not exist" Oct 31 00:37:54.377935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.377855 5577 scope.go:122] "RemoveContainer" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" Oct 31 00:37:54.378138 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.378120 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": container with ID starting with 5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72 not found: ID does not exist" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" Oct 31 00:37:54.378202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378148 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72"} err="failed to get container status \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": rpc error: code = NotFound desc = could not find container \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": container with ID starting with 5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72 not found: ID does not exist" Oct 31 00:37:54.378202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378165 5577 scope.go:122] "RemoveContainer" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" Oct 31 00:37:54.378440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378422 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8"} err="failed to get container status \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": rpc error: code = NotFound desc = could not find container \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": container with ID starting with f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8 not found: ID does not exist" Oct 31 00:37:54.378497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378440 5577 scope.go:122] "RemoveContainer" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" Oct 31 00:37:54.378667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378652 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921"} err="failed to get container status \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": rpc error: code = NotFound desc = could not find container \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": container with ID starting with 5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921 not found: ID does not exist" Oct 31 00:37:54.378744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378667 5577 scope.go:122] "RemoveContainer" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" Oct 31 00:37:54.378902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378886 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72"} err="failed to get container status \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": rpc error: code = NotFound desc = could not find container \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": container with ID starting with 5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72 not found: ID does not exist" Oct 31 00:37:54.378902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.378902 5577 scope.go:122] "RemoveContainer" containerID="f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8" Oct 31 00:37:54.379121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379100 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8"} err="failed to get container status \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": rpc error: code = NotFound desc = could not find container \"f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8\": container with ID starting with f44e7e83063d3eaffa6df60ec9d0dcb23abb28ac42309007854df7c757a82cf8 not found: ID does not exist" Oct 31 00:37:54.379187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379124 5577 scope.go:122] "RemoveContainer" containerID="5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921" Oct 31 00:37:54.379339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379321 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921"} err="failed to get container status \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": rpc error: code = NotFound desc = could not find container \"5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921\": container with ID starting with 5d4d427b9aa41781366de3fcc88d7b578b36d9ed38e11644998ae4e351297921 not found: ID does not exist" Oct 31 00:37:54.379446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379339 5577 scope.go:122] "RemoveContainer" containerID="5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72" Oct 31 00:37:54.379546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379531 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72"} err="failed to get container status \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": rpc error: code = NotFound desc = could not find container \"5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72\": container with ID starting with 5aadeb3332404257c91a851cbad203fe44ec5b6155d9c523e67bf5242645bd72 not found: ID does not exist" Oct 31 00:37:54.379590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.379547 5577 scope.go:122] "RemoveContainer" containerID="15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921" Oct 31 00:37:54.382649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.382626 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:54.387300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.387278 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:37:54.387615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.387602 5577 scope.go:122] "RemoveContainer" containerID="15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921" Oct 31 00:37:54.387974 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.387954 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921\": container with ID starting with 15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921 not found: ID does not exist" containerID="15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921" Oct 31 00:37:54.388057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.387978 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921"} err="failed to get container status \"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921\": rpc error: code = NotFound desc = could not find container \"15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921\": container with ID starting with 15d36042da253585fddcdd5b6ab001d7c1fe6894066894207c072c347ede5921 not found: ID does not exist" Oct 31 00:37:54.388057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.387993 5577 scope.go:122] "RemoveContainer" containerID="50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" Oct 31 00:37:54.395599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.395581 5577 scope.go:122] "RemoveContainer" containerID="191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" Oct 31 00:37:54.399140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.399115 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:37:54.403026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.403003 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:37:54.404116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.404100 5577 scope.go:122] "RemoveContainer" containerID="5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5" Oct 31 00:37:54.411530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.411514 5577 scope.go:122] "RemoveContainer" containerID="50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" Oct 31 00:37:54.411810 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.411790 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf\": container with ID starting with 50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf not found: ID does not exist" containerID="50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" Oct 31 00:37:54.411862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.411817 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf"} err="failed to get container status \"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf\": rpc error: code = NotFound desc = could not find container \"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf\": container with ID starting with 50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf not found: ID does not exist" Oct 31 00:37:54.411862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.411836 5577 scope.go:122] "RemoveContainer" containerID="191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" Oct 31 00:37:54.412112 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.412096 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1\": container with ID starting with 191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1 not found: ID does not exist" containerID="191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" Oct 31 00:37:54.412156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412119 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1"} err="failed to get container status \"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1\": rpc error: code = NotFound desc = could not find container \"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1\": container with ID starting with 191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1 not found: ID does not exist" Oct 31 00:37:54.412156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412136 5577 scope.go:122] "RemoveContainer" containerID="5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5" Oct 31 00:37:54.412343 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.412329 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5\": container with ID starting with 5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5 not found: ID does not exist" containerID="5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5" Oct 31 00:37:54.412390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412346 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5"} err="failed to get container status \"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5\": rpc error: code = NotFound desc = could not find container \"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5\": container with ID starting with 5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5 not found: ID does not exist" Oct 31 00:37:54.412390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412357 5577 scope.go:122] "RemoveContainer" containerID="50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf" Oct 31 00:37:54.412592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412577 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf"} err="failed to get container status \"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf\": rpc error: code = NotFound desc = could not find container \"50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf\": container with ID starting with 50b7e4aa07fa393e3bd816de2cba3af3f38946321014519d4f9bc543995fcadf not found: ID does not exist" Oct 31 00:37:54.412635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412593 5577 scope.go:122] "RemoveContainer" containerID="191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1" Oct 31 00:37:54.412801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412785 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1"} err="failed to get container status \"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1\": rpc error: code = NotFound desc = could not find container \"191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1\": container with ID starting with 191ad0ca77ac555538ee2ab5313bc353a2d4c449e953f087165b0bd02005cee1 not found: ID does not exist" Oct 31 00:37:54.412855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412801 5577 scope.go:122] "RemoveContainer" containerID="5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5" Oct 31 00:37:54.413016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.412997 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5"} err="failed to get container status \"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5\": rpc error: code = NotFound desc = could not find container \"5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5\": container with ID starting with 5bff4043ee063ffc33a4cf1bce98f95684b61500384b739a7e3cdedb115877e5 not found: ID does not exist" Oct 31 00:37:54.413058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.413017 5577 scope.go:122] "RemoveContainer" containerID="5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf" Oct 31 00:37:54.419412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.419381 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.419413 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.419429 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4e97df37-be64-4b72-a93b-9dbf940ca86b-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.419509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.419444 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4e97df37-be64-4b72-a93b-9dbf940ca86b-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:37:54.420496 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.420481 5577 scope.go:122] "RemoveContainer" containerID="5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf" Oct 31 00:37:54.420802 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:54.420782 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf\": container with ID starting with 5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf not found: ID does not exist" containerID="5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf" Oct 31 00:37:54.420864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.420809 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf"} err="failed to get container status \"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf\": rpc error: code = NotFound desc = could not find container \"5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf\": container with ID starting with 5022a83e7f2a51c758cb583832901aac2912f7291700bc99a8e253eee4b71ebf not found: ID does not exist" Oct 31 00:37:54.621925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.621893 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:37:54.628357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.628332 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:37:54.758242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:54.758205 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:37:55.129507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:55.129470 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" path="/var/lib/kubelet/pods/4e97df37-be64-4b72-a93b-9dbf940ca86b/volumes" Oct 31 00:37:55.130230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:55.130212 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" path="/var/lib/kubelet/pods/a7f8fa61-d4dd-4a65-9015-249bd2067703/volumes" Oct 31 00:37:55.130800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:55.130779 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" path="/var/lib/kubelet/pods/be23abc0-0155-4ed5-8c30-fd61fa5bde5e/volumes" Oct 31 00:37:55.131276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:55.131263 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1406343-3f9e-42b7-a136-a18849175421" path="/var/lib/kubelet/pods/e1406343-3f9e-42b7-a136-a18849175421/volumes" Oct 31 00:37:55.131627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:37:55.131615 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e99107d2-526a-494e-accd-e32ca14a5507" path="/var/lib/kubelet/pods/e99107d2-526a-494e-accd-e32ca14a5507/volumes" Oct 31 00:37:59.292546 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:37:59.292506 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:38:00.980408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:00.980376 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.053207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053175 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:38:01.053648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053628 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="kube-rbac-proxy" Oct 31 00:38:01.053690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053651 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="kube-rbac-proxy" Oct 31 00:38:01.053690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053669 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e1406343-3f9e-42b7-a136-a18849175421" containerName="otc-container" Oct 31 00:38:01.053690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053676 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1406343-3f9e-42b7-a136-a18849175421" containerName="otc-container" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053693 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="method-gate" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053701 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="method-gate" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053729 5577 cpu_manager.go:414] "containerMap: removing container" podUID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" containerName="perses" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053737 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" containerName="perses" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053747 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="config-reloader" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053755 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="config-reloader" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053766 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="kube-rbac-proxy" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053774 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="kube-rbac-proxy" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053784 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="method-gate" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053791 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="method-gate" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053808 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="alertmanager" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053816 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="alertmanager" Oct 31 00:38:01.053825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053828 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053835 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053854 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053861 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053870 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="init-config-reloader" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053877 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="init-config-reloader" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053957 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="alertmanager" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053969 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053980 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="prom-label-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053991 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="kube-rbac-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.053999 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1406343-3f9e-42b7-a136-a18849175421" containerName="otc-container" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.054008 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99107d2-526a-494e-accd-e32ca14a5507" containerName="method-gate" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.054015 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="be23abc0-0155-4ed5-8c30-fd61fa5bde5e" containerName="perses" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.054023 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e97df37-be64-4b72-a93b-9dbf940ca86b" containerName="config-reloader" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.054032 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="kube-rbac-proxy" Oct 31 00:38:01.054228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.054039 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7f8fa61-d4dd-4a65-9015-249bd2067703" containerName="method-gate" Oct 31 00:38:01.069650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.069623 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:38:01.069858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.069764 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.078064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.078019 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:38:01.079839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.079822 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:38:01.081355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.081317 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-l668c\"" Oct 31 00:38:01.081773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.081758 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.093001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.092976 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:38:01.115293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.115250 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.146004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.145974 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:38:01.146004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.145973 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-pxfpd\"" Oct 31 00:38:01.146200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146027 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:38:01.146366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146348 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:38:01.146484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146387 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:38:01.146484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146402 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:38:01.146484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146405 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:38:01.146601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.146510 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:38:01.177435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177408 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.177569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177441 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.177569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177465 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.177569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177483 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs6k4\" (UniqueName: \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.177691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177593 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.177691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177631 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.177691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177652 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.177828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.177749 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdl8s\" (UniqueName: \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.250825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.250788 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.278630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278600 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.278808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278636 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.278808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278763 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.278943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278804 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs6k4\" (UniqueName: \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.278943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278845 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.278943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278894 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.278943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278926 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.279134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.278957 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdl8s\" (UniqueName: \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.279356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.279334 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.279628 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.279595 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.279686 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.279636 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.279790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.279748 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.281345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.281320 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.281439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.281326 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.293796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.293670 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs6k4\" (UniqueName: \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\") pod \"data-science-thanos-querier-proxy-54fbc887bb-kswv5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.293796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.293777 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdl8s\" (UniqueName: \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vt9sh\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.380540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.380484 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.390395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.390368 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:38:01.400611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.400587 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:38:01.550409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.550324 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:38:01.554784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.554760 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.563121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.563089 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:38:01.567081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.567044 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:38:01.567391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.567161 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.568972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.568938 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:38:01.581730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.581688 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:38:01.582006 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.581988 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-d7w2x\"" Oct 31 00:38:01.582147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.582131 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:38:01.582410 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.582350 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:38:01.582520 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.582504 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:38:01.582636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.582537 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:38:01.583082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.583064 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:38:01.646627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.646599 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:38:01.683058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.682901 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lppv\" (UniqueName: \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.682957 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.683058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.682994 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683026 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683109 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683195 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683224 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.683448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683306 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.683448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683333 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6k2lp\" (UniqueName: \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.683810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.683792 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:38:01.687166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.687140 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:38:01.690180 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:38:01.690157 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf97e1bcf_cb1b_407a_a6eb_c14dd62faddc.slice/crio-c86ae700417e987e075e4332343099d329e5373f1071f1af1d4f78bbc6554fd3 WatchSource:0}: Error finding container c86ae700417e987e075e4332343099d329e5373f1071f1af1d4f78bbc6554fd3: Status 404 returned error can't find the container with id c86ae700417e987e075e4332343099d329e5373f1071f1af1d4f78bbc6554fd3 Oct 31 00:38:01.712924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.712891 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.784529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784499 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.784683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784549 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.784683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784661 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.784830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784699 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.784884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784844 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.784950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784900 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.784950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.784927 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6k2lp\" (UniqueName: \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.785051 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:38:01.784993 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:38:01.785051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.785022 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lppv\" (UniqueName: \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.785164 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:38:01.785066 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs podName:1514907a-4a96-42de-965e-1fcb967eeef4 nodeName:}" failed. No retries permitted until 2026-10-31 00:38:02.285044402 +0000 UTC m=+2321.276703987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:38:01.785164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.785121 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.785351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.785320 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.785678 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.785651 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.785778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.785692 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.787583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.787558 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.787675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.787596 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.788029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.788012 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.798447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.798416 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lppv\" (UniqueName: \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:01.798599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.798580 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6k2lp\" (UniqueName: \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:01.865019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.864983 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:01.998182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:01.998144 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.121427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.121316 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.288773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.288733 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:02.291560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.291535 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:02.342194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.342151 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerStarted","Data":"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78"} Oct 31 00:38:02.342396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.342271 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerStarted","Data":"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e"} Oct 31 00:38:02.342396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.342312 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerStarted","Data":"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2"} Oct 31 00:38:02.342396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.342345 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerStarted","Data":"1705b0f494ca7e5ffcfcf2948b772d436ead0e1df75c1428903d49e652205d6a"} Oct 31 00:38:02.344864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.344833 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerStarted","Data":"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c"} Oct 31 00:38:02.344985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.344883 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerStarted","Data":"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0"} Oct 31 00:38:02.344985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.344907 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerStarted","Data":"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c"} Oct 31 00:38:02.344985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.344934 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerStarted","Data":"c86ae700417e987e075e4332343099d329e5373f1071f1af1d4f78bbc6554fd3"} Oct 31 00:38:02.363757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.363674 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" podStartSLOduration=1.3636554379999999 podStartE2EDuration="1.363655438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:02.362138436 +0000 UTC m=+2321.353798042" watchObservedRunningTime="2026-10-31 00:38:02.363655438 +0000 UTC m=+2321.355315023" Oct 31 00:38:02.385158 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.385059 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" podStartSLOduration=1.385044375 podStartE2EDuration="1.385044375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:02.384547265 +0000 UTC m=+2321.376206858" watchObservedRunningTime="2026-10-31 00:38:02.385044375 +0000 UTC m=+2321.376703958" Oct 31 00:38:02.470536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.470500 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.472783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.472753 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:38:02.665341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.665253 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:02.760736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.760673 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:38:02.764585 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:38:02.764552 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1514907a_4a96_42de_965e_1fcb967eeef4.slice/crio-cbd5cb5b305c5221d49ec3daf73c92ec628ee2caff0164cdce13d9b86f24f5b4 WatchSource:0}: Error finding container cbd5cb5b305c5221d49ec3daf73c92ec628ee2caff0164cdce13d9b86f24f5b4: Status 404 returned error can't find the container with id cbd5cb5b305c5221d49ec3daf73c92ec628ee2caff0164cdce13d9b86f24f5b4 Oct 31 00:38:02.767149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.767105 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:02.767252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:02.767189 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:38:03.350941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:03.350906 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1514907a-4a96-42de-965e-1fcb967eeef4","Type":"ContainerStarted","Data":"64f23629542a910a0fed12b39d7af3ab0ca04bf33452f6b0cefde2787740d90f"} Oct 31 00:38:03.351440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:03.350960 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1514907a-4a96-42de-965e-1fcb967eeef4","Type":"ContainerStarted","Data":"cbd5cb5b305c5221d49ec3daf73c92ec628ee2caff0164cdce13d9b86f24f5b4"} Oct 31 00:38:03.373398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:03.373338 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.373320866 podStartE2EDuration="2.373320866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:03.372638571 +0000 UTC m=+2322.364298159" watchObservedRunningTime="2026-10-31 00:38:03.373320866 +0000 UTC m=+2322.364980451" Oct 31 00:38:07.120645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.120538 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:07.343279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.343238 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:07.399986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.399907 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:38:07.403830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.403805 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.417033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.417002 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:38:07.467896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.467863 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:07.479642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.479606 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:38:07.479825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.479689 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:38:07.479825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.479691 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:38:07.479825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.479727 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-94cpx\"" Oct 31 00:38:07.479825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.479700 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:38:07.537160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537123 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537184 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbpm\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537274 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537328 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537418 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537441 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.537506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.537464 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638478 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbpm\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638524 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638549 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638570 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638588 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638608 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.638812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638653 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.639090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.638992 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.641685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.641654 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.641838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.641749 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.641928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.641903 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.642022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.642004 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.642111 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.642091 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.651432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.651385 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbpm\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.726104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.726061 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:07.817504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:07.817467 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:08.015696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:08.015606 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:38:08.023214 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:38:08.023173 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9c8db9_606f_408d_846e_2ba362895511.slice/crio-80843f7e08c277d96ecddcf6eb798e70c3636b532cd797023c65139b9726438f WatchSource:0}: Error finding container 80843f7e08c277d96ecddcf6eb798e70c3636b532cd797023c65139b9726438f: Status 404 returned error can't find the container with id 80843f7e08c277d96ecddcf6eb798e70c3636b532cd797023c65139b9726438f Oct 31 00:38:08.379685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:08.379581 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerStarted","Data":"80843f7e08c277d96ecddcf6eb798e70c3636b532cd797023c65139b9726438f"} Oct 31 00:38:09.385092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:09.385056 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerStarted","Data":"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251"} Oct 31 00:38:09.961625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:09.961589 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:09.968612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:09.968582 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.062905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.062869 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.063084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.063009 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.066221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.066195 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.158793 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.158764 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.177851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.177816 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.282940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.282884 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:10.558333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:10.558246 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:38:10.566768 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:38:10.566674 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5b2d78e_881b_4d63_8a03_33cb68b8ac02.slice/crio-905876c78ed8249940b18edbabbd21d69b2a9cdffb40d9ca44aa7a11938cb6a2 WatchSource:0}: Error finding container 905876c78ed8249940b18edbabbd21d69b2a9cdffb40d9ca44aa7a11938cb6a2: Status 404 returned error can't find the container with id 905876c78ed8249940b18edbabbd21d69b2a9cdffb40d9ca44aa7a11938cb6a2 Oct 31 00:38:11.395820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:11.395778 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5b2d78e-881b-4d63-8a03-33cb68b8ac02","Type":"ContainerStarted","Data":"764fcf26dfdbf6af09b7205b669e790fdd54ebd63c651249f9541abf0920b9bb"} Oct 31 00:38:11.396032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:11.395829 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5b2d78e-881b-4d63-8a03-33cb68b8ac02","Type":"ContainerStarted","Data":"905876c78ed8249940b18edbabbd21d69b2a9cdffb40d9ca44aa7a11938cb6a2"} Oct 31 00:38:11.396032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:11.395953 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:11.415841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:11.415779 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=10.415759286 podStartE2EDuration="10.415759286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:11.414518154 +0000 UTC m=+2330.406177751" watchObservedRunningTime="2026-10-31 00:38:11.415759286 +0000 UTC m=+2330.407418872" Oct 31 00:38:12.402243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:12.402204 5577 generic.go:356] "Generic (PLEG): container finished" podID="0f9c8db9-606f-408d-846e-2ba362895511" containerID="4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251" exitCode=0 Oct 31 00:38:12.402624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:12.402280 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerDied","Data":"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251"} Oct 31 00:38:13.410077 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:13.410041 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerStarted","Data":"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000"} Oct 31 00:38:13.410450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:13.410110 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerStarted","Data":"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a"} Oct 31 00:38:13.410450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:13.410318 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:13.436838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:13.436786 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=2.737752138 podStartE2EDuration="6.436772494s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.699020356s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:13.435033382 +0000 UTC m=+2332.426692966" watchObservedRunningTime="2026-10-31 00:38:13.436772494 +0000 UTC m=+2332.428432077" Oct 31 00:38:19.415670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:19.415634 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:38:19.755270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:19.755236 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:19.926857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:19.926810 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:20.057504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:20.057413 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:22.404208 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:22.404177 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:38:25.689035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:25.689001 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:25.822821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:25.822780 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.165314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:26.165275 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.303815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:26.303778 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:26.633964 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:26.633921 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:29.293123 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:38:29.293044 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:38:40.526125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:40.526083 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:46.579800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:46.579761 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:46.708931 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:46.708866 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:48.684234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:48.684190 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:48.829749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:48.829692 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:49.047422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:49.047389 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.397014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:52.396963 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:52.518124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:52.518084 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:53.193923 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:53.193878 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:53.319749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:53.319686 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:53.462352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:53.462240 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:53.591020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:53.590966 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:58.202868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:58.202835 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:58.330342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:58.330313 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:59.292468 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:38:59.292429 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:38:59.764481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:59.764449 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:38:59.908114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:38:59.908076 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:00.036669 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:00.036590 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:00.200905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:00.200866 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:03.906390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:03.906359 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.037264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:04.037232 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.345154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:04.345102 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.475098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:04.475055 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:04.606084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:04.605999 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.543196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:09.543159 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:09.667883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:09.667850 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:10.755588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:10.755543 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:10.945572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:10.945530 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:11.076489 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:11.076397 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:15.233922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:15.233883 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:15.367163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:15.367127 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.147759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:17.147721 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.277139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:17.277106 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:17.406293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:17.406194 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:20.966423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:20.966388 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:21.094645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.094616 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:21.372168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.372137 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:39:21.377341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.377317 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:39:21.552301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.552259 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:21.677049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.676957 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:21.830271 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:21.830237 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:26.653542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:26.653502 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:26.789811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:26.789769 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:26.924282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:26.924170 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:29.292341 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:29.292301 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:39:32.055395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.055322 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:39:32.055972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.055632 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="kube-rbac-proxy" containerID="cri-o://86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" gracePeriod=30 Oct 31 00:39:32.055972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.055936 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="method-gate" containerID="cri-o://2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" gracePeriod=30 Oct 31 00:39:32.056144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.055952 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="prom-label-proxy" containerID="cri-o://05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" gracePeriod=30 Oct 31 00:39:32.057177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.057156 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:39:32.057458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.057432 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="kube-rbac-proxy" containerID="cri-o://f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" gracePeriod=30 Oct 31 00:39:32.057625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.057557 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="prom-label-proxy" containerID="cri-o://b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" gracePeriod=30 Oct 31 00:39:32.057625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.057584 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="method-gate" containerID="cri-o://ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" gracePeriod=30 Oct 31 00:39:32.269913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.269819 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.280502 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.280420 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:39:32.280779 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.280752 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" containerID="cri-o://7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a" gracePeriod=120 Oct 31 00:39:32.281133 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.280978 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="config-reloader" containerID="cri-o://b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000" gracePeriod=120 Oct 31 00:39:32.371354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.371331 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:39:32.380649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.380627 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:39:32.415657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.415620 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.432399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.432360 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:39:32.432662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.432634 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1514907a-4a96-42de-965e-1fcb967eeef4" containerName="otc-container" containerID="cri-o://64f23629542a910a0fed12b39d7af3ab0ca04bf33452f6b0cefde2787740d90f" gracePeriod=30 Oct 31 00:39:32.464792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464751 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\") pod \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " Oct 31 00:39:32.464946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464813 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\" (UniqueName: \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\") pod \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " Oct 31 00:39:32.464946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464843 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\" (UniqueName: \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\") pod \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " Oct 31 00:39:32.464946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464868 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\" (UniqueName: \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\") pod \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " Oct 31 00:39:32.464946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464898 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\") pod \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " Oct 31 00:39:32.464946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464921 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\") pod \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\" (UID: \"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc\") " Oct 31 00:39:32.465227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.464986 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\" (UniqueName: \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\") pod \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " Oct 31 00:39:32.465227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.465027 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\" (UniqueName: \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\") pod \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\" (UID: \"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5\") " Oct 31 00:39:32.465227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.465144 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca" pod "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" (UID: "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.465432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.465311 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.465566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.465471 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca" pod "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" (UID: "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.465801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.465779 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config" pod "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" (UID: "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.466466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.466428 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config" pod "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" (UID: "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.469187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.469166 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4" pod "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" (UID: "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5"). InnerVolumeSpecName "kube-api-access-vs6k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.470556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.470534 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls" pod "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" (UID: "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.471018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.470859 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s" pod "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" (UID: "f97e1bcf-cb1b-407a-a6eb-c14dd62faddc"). InnerVolumeSpecName "kube-api-access-bdl8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.471018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.470895 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls" pod "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" (UID: "9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.494639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.494605 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:39:32.494890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.494865 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" containerName="perses" containerID="cri-o://764fcf26dfdbf6af09b7205b669e790fdd54ebd63c651249f9541abf0920b9bb" gracePeriod=30 Oct 31 00:39:32.551126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.551076 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566599 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs6k4\" (UniqueName: \"kubernetes.io/projected/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-kube-api-access-vs6k4\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566626 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdl8s\" (UniqueName: \"kubernetes.io/projected/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-kube-api-access-bdl8s\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566637 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566646 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566655 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566663 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.566630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.566671 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.684190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.684146 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.751858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.751818 5577 generic.go:356] "Generic (PLEG): container finished" podID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" exitCode=0 Oct 31 00:39:32.751858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.751858 5577 generic.go:356] "Generic (PLEG): container finished" podID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" exitCode=0 Oct 31 00:39:32.752126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.751868 5577 generic.go:356] "Generic (PLEG): container finished" podID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" exitCode=0 Oct 31 00:39:32.752126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.752107 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" Oct 31 00:39:32.753216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.753190 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerDied","Data":"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c"} Oct 31 00:39:32.753363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.753264 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerDied","Data":"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0"} Oct 31 00:39:32.753363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.753321 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerDied","Data":"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c"} Oct 31 00:39:32.753363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.753356 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh" event={"ID":"f97e1bcf-cb1b-407a-a6eb-c14dd62faddc","Type":"ContainerDied","Data":"c86ae700417e987e075e4332343099d329e5373f1071f1af1d4f78bbc6554fd3"} Oct 31 00:39:32.753532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.753399 5577 scope.go:122] "RemoveContainer" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" Oct 31 00:39:32.755398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.755374 5577 generic.go:356] "Generic (PLEG): container finished" podID="1514907a-4a96-42de-965e-1fcb967eeef4" containerID="64f23629542a910a0fed12b39d7af3ab0ca04bf33452f6b0cefde2787740d90f" exitCode=0 Oct 31 00:39:32.755507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.755483 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1514907a-4a96-42de-965e-1fcb967eeef4","Type":"ContainerDied","Data":"64f23629542a910a0fed12b39d7af3ab0ca04bf33452f6b0cefde2787740d90f"} Oct 31 00:39:32.758051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.758026 5577 generic.go:356] "Generic (PLEG): container finished" podID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" containerID="764fcf26dfdbf6af09b7205b669e790fdd54ebd63c651249f9541abf0920b9bb" exitCode=0 Oct 31 00:39:32.758303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.758281 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5b2d78e-881b-4d63-8a03-33cb68b8ac02","Type":"ContainerDied","Data":"764fcf26dfdbf6af09b7205b669e790fdd54ebd63c651249f9541abf0920b9bb"} Oct 31 00:39:32.760911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.760875 5577 generic.go:356] "Generic (PLEG): container finished" podID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" exitCode=0 Oct 31 00:39:32.760911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.760902 5577 generic.go:356] "Generic (PLEG): container finished" podID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" exitCode=0 Oct 31 00:39:32.760911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.760912 5577 generic.go:356] "Generic (PLEG): container finished" podID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" exitCode=0 Oct 31 00:39:32.761138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.760969 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerDied","Data":"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78"} Oct 31 00:39:32.761138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.761023 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerDied","Data":"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e"} Oct 31 00:39:32.761138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.761062 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerDied","Data":"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2"} Oct 31 00:39:32.761138 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.761099 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" event={"ID":"9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5","Type":"ContainerDied","Data":"1705b0f494ca7e5ffcfcf2948b772d436ead0e1df75c1428903d49e652205d6a"} Oct 31 00:39:32.761359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.761214 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5" Oct 31 00:39:32.768423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.766193 5577 scope.go:122] "RemoveContainer" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" Oct 31 00:39:32.771651 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.771533 5577 generic.go:356] "Generic (PLEG): container finished" podID="0f9c8db9-606f-408d-846e-2ba362895511" containerID="b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000" exitCode=0 Oct 31 00:39:32.771651 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.771569 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerDied","Data":"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000"} Oct 31 00:39:32.782481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.782446 5577 scope.go:122] "RemoveContainer" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" Oct 31 00:39:32.789736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.789694 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:39:32.795039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.795019 5577 scope.go:122] "RemoveContainer" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" Oct 31 00:39:32.795963 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.795824 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": container with ID starting with 05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c not found: ID does not exist" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" Oct 31 00:39:32.795963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.795867 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c"} err="failed to get container status \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": rpc error: code = NotFound desc = could not find container \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": container with ID starting with 05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c not found: ID does not exist" Oct 31 00:39:32.795963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.795893 5577 scope.go:122] "RemoveContainer" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" Oct 31 00:39:32.796211 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.796183 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": container with ID starting with 2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0 not found: ID does not exist" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" Oct 31 00:39:32.796333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796219 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0"} err="failed to get container status \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": rpc error: code = NotFound desc = could not find container \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": container with ID starting with 2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0 not found: ID does not exist" Oct 31 00:39:32.796333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796238 5577 scope.go:122] "RemoveContainer" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" Oct 31 00:39:32.796449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796372 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vt9sh"] Oct 31 00:39:32.796547 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.796528 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": container with ID starting with 86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c not found: ID does not exist" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" Oct 31 00:39:32.796610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796557 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c"} err="failed to get container status \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": rpc error: code = NotFound desc = could not find container \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": container with ID starting with 86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c not found: ID does not exist" Oct 31 00:39:32.796610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796580 5577 scope.go:122] "RemoveContainer" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" Oct 31 00:39:32.796898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796872 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c"} err="failed to get container status \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": rpc error: code = NotFound desc = could not find container \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": container with ID starting with 05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c not found: ID does not exist" Oct 31 00:39:32.796981 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.796899 5577 scope.go:122] "RemoveContainer" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" Oct 31 00:39:32.797170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797151 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0"} err="failed to get container status \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": rpc error: code = NotFound desc = could not find container \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": container with ID starting with 2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0 not found: ID does not exist" Oct 31 00:39:32.797233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797171 5577 scope.go:122] "RemoveContainer" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" Oct 31 00:39:32.797446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797430 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c"} err="failed to get container status \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": rpc error: code = NotFound desc = could not find container \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": container with ID starting with 86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c not found: ID does not exist" Oct 31 00:39:32.797521 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797448 5577 scope.go:122] "RemoveContainer" containerID="05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c" Oct 31 00:39:32.797837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797814 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c"} err="failed to get container status \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": rpc error: code = NotFound desc = could not find container \"05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c\": container with ID starting with 05c72ad141b124fdf48c008bd7689beed198833044010a61eb0efc10a63a7a1c not found: ID does not exist" Oct 31 00:39:32.797940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.797838 5577 scope.go:122] "RemoveContainer" containerID="2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0" Oct 31 00:39:32.798077 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.798048 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0"} err="failed to get container status \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": rpc error: code = NotFound desc = could not find container \"2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0\": container with ID starting with 2c05535245f21282ee49768a93f200cea1b5ba620061839b9a6dd26a72db51d0 not found: ID does not exist" Oct 31 00:39:32.798150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.798078 5577 scope.go:122] "RemoveContainer" containerID="86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c" Oct 31 00:39:32.798417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.798391 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c"} err="failed to get container status \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": rpc error: code = NotFound desc = could not find container \"86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c\": container with ID starting with 86d4dd715e7a8bf2a35ab70a4d2ee3bc2892e5b794c5a77071cd8efbbb3d415c not found: ID does not exist" Oct 31 00:39:32.798417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.798417 5577 scope.go:122] "RemoveContainer" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" Oct 31 00:39:32.803001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.802978 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:39:32.808154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.808137 5577 scope.go:122] "RemoveContainer" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" Oct 31 00:39:32.808226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.808213 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:39:32.812570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.812541 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:39:32.813621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.813600 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-kswv5"] Oct 31 00:39:32.816599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.816579 5577 scope.go:122] "RemoveContainer" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" Oct 31 00:39:32.820741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.820700 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:32.826049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826017 5577 scope.go:122] "RemoveContainer" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" Oct 31 00:39:32.826323 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.826299 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": container with ID starting with b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78 not found: ID does not exist" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" Oct 31 00:39:32.826382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826329 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78"} err="failed to get container status \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": rpc error: code = NotFound desc = could not find container \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": container with ID starting with b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78 not found: ID does not exist" Oct 31 00:39:32.826382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826344 5577 scope.go:122] "RemoveContainer" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" Oct 31 00:39:32.826611 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.826594 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": container with ID starting with ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e not found: ID does not exist" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" Oct 31 00:39:32.826664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826613 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e"} err="failed to get container status \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": rpc error: code = NotFound desc = could not find container \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": container with ID starting with ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e not found: ID does not exist" Oct 31 00:39:32.826664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826624 5577 scope.go:122] "RemoveContainer" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" Oct 31 00:39:32.826890 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:32.826874 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": container with ID starting with f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2 not found: ID does not exist" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" Oct 31 00:39:32.826959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826894 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2"} err="failed to get container status \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": rpc error: code = NotFound desc = could not find container \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": container with ID starting with f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2 not found: ID does not exist" Oct 31 00:39:32.826959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.826910 5577 scope.go:122] "RemoveContainer" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" Oct 31 00:39:32.827178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827160 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78"} err="failed to get container status \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": rpc error: code = NotFound desc = could not find container \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": container with ID starting with b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78 not found: ID does not exist" Oct 31 00:39:32.827221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827179 5577 scope.go:122] "RemoveContainer" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" Oct 31 00:39:32.827413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827396 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e"} err="failed to get container status \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": rpc error: code = NotFound desc = could not find container \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": container with ID starting with ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e not found: ID does not exist" Oct 31 00:39:32.827452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827415 5577 scope.go:122] "RemoveContainer" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" Oct 31 00:39:32.827650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827628 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2"} err="failed to get container status \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": rpc error: code = NotFound desc = could not find container \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": container with ID starting with f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2 not found: ID does not exist" Oct 31 00:39:32.827818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827651 5577 scope.go:122] "RemoveContainer" containerID="b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78" Oct 31 00:39:32.827907 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827891 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78"} err="failed to get container status \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": rpc error: code = NotFound desc = could not find container \"b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78\": container with ID starting with b77fbe78c00308061aad1d6ca74a2a14c6ab21fdfb62fc8c17314021d77a8a78 not found: ID does not exist" Oct 31 00:39:32.827949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.827907 5577 scope.go:122] "RemoveContainer" containerID="ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e" Oct 31 00:39:32.828124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.828109 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e"} err="failed to get container status \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": rpc error: code = NotFound desc = could not find container \"ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e\": container with ID starting with ceb3bb8be61235ae6d3f6f7893a161bf2caf63df1d3b8f37b707e4f7bde77a6e not found: ID does not exist" Oct 31 00:39:32.828124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.828124 5577 scope.go:122] "RemoveContainer" containerID="f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2" Oct 31 00:39:32.828332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.828314 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2"} err="failed to get container status \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": rpc error: code = NotFound desc = could not find container \"f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2\": container with ID starting with f8efa4209e90cf49a1e1c1ee56c98354c62e5c032490ece6eb43adffe14885f2 not found: ID does not exist" Oct 31 00:39:32.869492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869412 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869470 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869532 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\" (UniqueName: \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\") pod \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " Oct 31 00:39:32.869695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869576 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\" (UniqueName: \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869612 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869647 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\" (UniqueName: \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\") pod \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " Oct 31 00:39:32.869961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869761 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " Oct 31 00:39:32.869961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869808 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869837 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") pod \"1514907a-4a96-42de-965e-1fcb967eeef4\" (UID: \"1514907a-4a96-42de-965e-1fcb967eeef4\") " Oct 31 00:39:32.869961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869865 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\") pod \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\" (UID: \"d5b2d78e-881b-4d63-8a03-33cb68b8ac02\") " Oct 31 00:39:32.869961 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869933 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config" pod "d5b2d78e-881b-4d63-8a03-33cb68b8ac02" (UID: "d5b2d78e-881b-4d63-8a03-33cb68b8ac02"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.870220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.869969 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:39:32.870274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.870258 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1514907a-4a96-42de-965e-1fcb967eeef4-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.870321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.870279 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.873053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873019 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.873053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873043 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.873202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873044 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp" pod "d5b2d78e-881b-4d63-8a03-33cb68b8ac02" (UID: "d5b2d78e-881b-4d63-8a03-33cb68b8ac02"). InnerVolumeSpecName "kube-api-access-6k2lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.873202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873063 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.873283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873256 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "kube-api-access-5lppv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:32.873956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.873939 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "d5b2d78e-881b-4d63-8a03-33cb68b8ac02" (UID: "d5b2d78e-881b-4d63-8a03-33cb68b8ac02"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:39:32.874195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.874172 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs" pod "1514907a-4a96-42de-965e-1fcb967eeef4" (UID: "1514907a-4a96-42de-965e-1fcb967eeef4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:32.970792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970754 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lppv\" (UniqueName: \"kubernetes.io/projected/1514907a-4a96-42de-965e-1fcb967eeef4-kube-api-access-5lppv\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.970792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970793 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.970986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970808 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6k2lp\" (UniqueName: \"kubernetes.io/projected/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-kube-api-access-6k2lp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.970986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970846 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:39:32.970986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970861 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.970986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970877 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.970986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.970891 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1514907a-4a96-42de-965e-1fcb967eeef4-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:32.983671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:32.983650 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:39:33.024873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.024834 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins" pod "d5b2d78e-881b-4d63-8a03-33cb68b8ac02" (UID: "d5b2d78e-881b-4d63-8a03-33cb68b8ac02"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:33.071550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.071519 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:39:33.071920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.071556 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/d5b2d78e-881b-4d63-8a03-33cb68b8ac02-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:33.130313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.130222 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" path="/var/lib/kubelet/pods/9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5/volumes" Oct 31 00:39:33.130806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.130786 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" path="/var/lib/kubelet/pods/f97e1bcf-cb1b-407a-a6eb-c14dd62faddc/volumes" Oct 31 00:39:33.777586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.777546 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1514907a-4a96-42de-965e-1fcb967eeef4","Type":"ContainerDied","Data":"cbd5cb5b305c5221d49ec3daf73c92ec628ee2caff0164cdce13d9b86f24f5b4"} Oct 31 00:39:33.777813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.777556 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:39:33.777813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.777614 5577 scope.go:122] "RemoveContainer" containerID="64f23629542a910a0fed12b39d7af3ab0ca04bf33452f6b0cefde2787740d90f" Oct 31 00:39:33.778980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.778964 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:39:33.779049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.778978 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"d5b2d78e-881b-4d63-8a03-33cb68b8ac02","Type":"ContainerDied","Data":"905876c78ed8249940b18edbabbd21d69b2a9cdffb40d9ca44aa7a11938cb6a2"} Oct 31 00:39:33.786164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.786149 5577 scope.go:122] "RemoveContainer" containerID="764fcf26dfdbf6af09b7205b669e790fdd54ebd63c651249f9541abf0920b9bb" Oct 31 00:39:33.797674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.797650 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:39:33.805383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.805362 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:39:33.817162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.817132 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:39:33.820970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:33.820945 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:39:34.415183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:34.415141 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.101:9093/-/ready\": dial tcp 10.133.0.101:9093: connect: connection refused" Oct 31 00:39:34.500872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:34.500837 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:35.134842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:35.134797 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1514907a-4a96-42de-965e-1fcb967eeef4" path="/var/lib/kubelet/pods/1514907a-4a96-42de-965e-1fcb967eeef4/volumes" Oct 31 00:39:35.135352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:35.135335 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" path="/var/lib/kubelet/pods/d5b2d78e-881b-4d63-8a03-33cb68b8ac02/volumes" Oct 31 00:39:39.415173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:39.415137 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.101:9093/-/ready\": dial tcp 10.133.0.101:9093: connect: connection refused" Oct 31 00:39:42.521936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.521909 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:39:42.653294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653206 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653254 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653334 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653358 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653383 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653420 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653499 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\") pod \"0f9c8db9-606f-408d-846e-2ba362895511\" (UID: \"0f9c8db9-606f-408d-846e-2ba362895511\") " Oct 31 00:39:42.653920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.653884 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:42.655681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.655657 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "kube-api-access-whbpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:42.655987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.655967 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.656232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.656209 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:39:42.656295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.656245 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:42.659642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.659616 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.667117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.667090 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config" pod "0f9c8db9-606f-408d-846e-2ba362895511" (UID: "0f9c8db9-606f-408d-846e-2ba362895511"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:39:42.754252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754213 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754243 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754253 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0f9c8db9-606f-408d-846e-2ba362895511-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754264 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbpm\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-kube-api-access-whbpm\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754273 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754282 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0f9c8db9-606f-408d-846e-2ba362895511-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.754525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.754291 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/0f9c8db9-606f-408d-846e-2ba362895511-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:42.823083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.823046 5577 generic.go:356] "Generic (PLEG): container finished" podID="0f9c8db9-606f-408d-846e-2ba362895511" containerID="7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a" exitCode=0 Oct 31 00:39:42.823285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.823108 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerDied","Data":"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a"} Oct 31 00:39:42.823285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.823114 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 31 00:39:42.823285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.823157 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"0f9c8db9-606f-408d-846e-2ba362895511","Type":"ContainerDied","Data":"80843f7e08c277d96ecddcf6eb798e70c3636b532cd797023c65139b9726438f"} Oct 31 00:39:42.823285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.823183 5577 scope.go:122] "RemoveContainer" containerID="b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000" Oct 31 00:39:42.832535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.832459 5577 scope.go:122] "RemoveContainer" containerID="7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a" Oct 31 00:39:42.841832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.841809 5577 scope.go:122] "RemoveContainer" containerID="4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251" Oct 31 00:39:42.845750 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.845723 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:39:42.850210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850188 5577 scope.go:122] "RemoveContainer" containerID="b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000" Oct 31 00:39:42.850274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850247 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 31 00:39:42.850462 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:42.850446 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000\": container with ID starting with b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000 not found: ID does not exist" containerID="b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000" Oct 31 00:39:42.850508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850470 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000"} err="failed to get container status \"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000\": rpc error: code = NotFound desc = could not find container \"b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000\": container with ID starting with b4e3b8d176e410ea5f8f1ba2d3cb8c41309da7f90cdd1a94b6e25cbbf5324000 not found: ID does not exist" Oct 31 00:39:42.850508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850487 5577 scope.go:122] "RemoveContainer" containerID="7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a" Oct 31 00:39:42.850735 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:42.850718 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a\": container with ID starting with 7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a not found: ID does not exist" containerID="7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a" Oct 31 00:39:42.850785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850740 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a"} err="failed to get container status \"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a\": rpc error: code = NotFound desc = could not find container \"7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a\": container with ID starting with 7ba3be7305882ce5d6234ebad80bf2200b4ba582517d84b3ca00f6e8aa224c8a not found: ID does not exist" Oct 31 00:39:42.850785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.850757 5577 scope.go:122] "RemoveContainer" containerID="4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251" Oct 31 00:39:42.851009 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:42.850995 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251\": container with ID starting with 4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251 not found: ID does not exist" containerID="4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251" Oct 31 00:39:42.851049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:42.851012 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251"} err="failed to get container status \"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251\": rpc error: code = NotFound desc = could not find container \"4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251\": container with ID starting with 4c2746570bf074d45e6600060d7c9326b5a270d1d6c28f5ee0009c4e7e36c251 not found: ID does not exist" Oct 31 00:39:43.129533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:43.129498 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9c8db9-606f-408d-846e-2ba362895511" path="/var/lib/kubelet/pods/0f9c8db9-606f-408d-846e-2ba362895511/volumes" Oct 31 00:39:45.837850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:45.837817 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:46.595658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.595618 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:39:46.596041 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596020 5577 cpu_manager.go:414] "containerMap: removing container" podUID="1514907a-4a96-42de-965e-1fcb967eeef4" containerName="otc-container" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596042 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1514907a-4a96-42de-965e-1fcb967eeef4" containerName="otc-container" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596057 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="method-gate" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596064 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="method-gate" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596077 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="prom-label-proxy" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596085 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="prom-label-proxy" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596097 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" containerName="perses" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" containerName="perses" Oct 31 00:39:46.596134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596132 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596139 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596149 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="init-config-reloader" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596156 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="init-config-reloader" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596166 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596173 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596184 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596191 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596200 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="config-reloader" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596208 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="config-reloader" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596224 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="prom-label-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596231 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="prom-label-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596242 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="method-gate" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596249 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="method-gate" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596327 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5b2d78e-881b-4d63-8a03-33cb68b8ac02" containerName="perses" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596373 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596383 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="config-reloader" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596392 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9c8db9-606f-408d-846e-2ba362895511" containerName="alertmanager" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596417 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="kube-rbac-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596427 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="prom-label-proxy" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596435 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="method-gate" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596446 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="1514907a-4a96-42de-965e-1fcb967eeef4" containerName="otc-container" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596453 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f97e1bcf-cb1b-407a-a6eb-c14dd62faddc" containerName="method-gate" Oct 31 00:39:46.596532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.596461 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9769fd5d-a0d6-44e8-8dfd-68d9c0ff7ef5" containerName="prom-label-proxy" Oct 31 00:39:46.599743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.599723 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:46.606163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.605951 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:39:46.606734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.606595 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:39:46.606734 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.606682 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:39:46.606969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.606948 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:39:46.685849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.685814 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vp96\" (UniqueName: \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\") pod \"seaweedfs-bucket-creator\" (UID: \"b1046857-f419-4b53-b1c2-7b2c20237b77\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:46.786453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.786416 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vp96\" (UniqueName: \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\") pod \"seaweedfs-bucket-creator\" (UID: \"b1046857-f419-4b53-b1c2-7b2c20237b77\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:46.799233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.799206 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vp96\" (UniqueName: \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\") pod \"seaweedfs-bucket-creator\" (UID: \"b1046857-f419-4b53-b1c2-7b2c20237b77\") " pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:46.911679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:46.911598 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:47.181149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:47.181060 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:39:47.184277 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:39:47.184248 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1046857_f419_4b53_b1c2_7b2c20237b77.slice/crio-f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6 WatchSource:0}: Error finding container f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6: Status 404 returned error can't find the container with id f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6 Oct 31 00:39:47.846861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:47.846825 5577 generic.go:356] "Generic (PLEG): container finished" podID="b1046857-f419-4b53-b1c2-7b2c20237b77" containerID="d08089ae055a228ffc859eab34c08c8de623c26e7112952da433f00fc6788e2a" exitCode=0 Oct 31 00:39:47.847019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:47.846907 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"b1046857-f419-4b53-b1c2-7b2c20237b77","Type":"ContainerDied","Data":"d08089ae055a228ffc859eab34c08c8de623c26e7112952da433f00fc6788e2a"} Oct 31 00:39:47.847019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:47.846947 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"b1046857-f419-4b53-b1c2-7b2c20237b77","Type":"ContainerStarted","Data":"f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6"} Oct 31 00:39:48.980086 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:48.980053 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:49.105331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.105242 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\" (UniqueName: \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\") pod \"b1046857-f419-4b53-b1c2-7b2c20237b77\" (UID: \"b1046857-f419-4b53-b1c2-7b2c20237b77\") " Oct 31 00:39:49.107421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.107397 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96" pod "b1046857-f419-4b53-b1c2-7b2c20237b77" (UID: "b1046857-f419-4b53-b1c2-7b2c20237b77"). InnerVolumeSpecName "kube-api-access-9vp96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:39:49.205809 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.205776 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vp96\" (UniqueName: \"kubernetes.io/projected/b1046857-f419-4b53-b1c2-7b2c20237b77-kube-api-access-9vp96\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:39:49.856132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.856101 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" Oct 31 00:39:49.856132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.856113 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/seaweedfs-bucket-creator" event={"ID":"b1046857-f419-4b53-b1c2-7b2c20237b77","Type":"ContainerDied","Data":"f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6"} Oct 31 00:39:49.856399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:49.856146 5577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f16b4852666b6888687d0002308fa8a60319e3d5bc3eb23f224247b6abe122e6" Oct 31 00:39:53.406783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.406746 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:39:53.407194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.407177 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b1046857-f419-4b53-b1c2-7b2c20237b77" containerName="create-bucket" Oct 31 00:39:53.407235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.407196 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1046857-f419-4b53-b1c2-7b2c20237b77" containerName="create-bucket" Oct 31 00:39:53.407270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.407245 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1046857-f419-4b53-b1c2-7b2c20237b77" containerName="create-bucket" Oct 31 00:39:53.410225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.410209 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.414016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.413993 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:39:53.414916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.414886 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 31 00:39:53.415028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.414918 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:39:53.415114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.415095 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 31 00:39:53.416014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.415988 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-grpc\"" Oct 31 00:39:53.416014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.416000 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-distributor-http\"" Oct 31 00:39:53.416130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.416041 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-btcx5\"" Oct 31 00:39:53.419543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.419521 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:39:53.491479 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.491445 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:53.539249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.539216 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.539446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.539253 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.539446 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.539384 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.539585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.539487 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hv9b\" (UniqueName: \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.539585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.539546 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.599472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.599443 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:39:53.602844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.602823 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.612564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.612541 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 31 00:39:53.612564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.612556 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 31 00:39:53.612564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.612545 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-usage-logs-loki-secret\"" Oct 31 00:39:53.615462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.615441 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:39:53.640816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.640782 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.640987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.640845 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hv9b\" (UniqueName: \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.640987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.640868 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.640987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.640915 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.641178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.641019 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.641735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.641598 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.641856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.641824 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.642405 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.642376 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:53.643901 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.643873 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.644130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.644105 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.656621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.656595 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hv9b\" (UniqueName: \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.663933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.663865 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:39:53.668029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.668011 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.676799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.676762 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:39:53.712522 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.712489 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-grpc\"" Oct 31 00:39:53.712690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.712488 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-http\"" Oct 31 00:39:53.721692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.721666 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:53.742962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.742932 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.743099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.742979 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.743099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.743087 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.743212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.743116 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.743212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.743166 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.743326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.743232 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft5th\" (UniqueName: \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.794424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.794316 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:53.844542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844511 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.844704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844555 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft5th\" (UniqueName: \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.844704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844622 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.844704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844685 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.844863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844824 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.844916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844863 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.844978 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844936 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.845035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.844997 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.845088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.845040 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.845088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.845073 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6srw\" (UniqueName: \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.845194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.845109 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.845629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.845606 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.846059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.846036 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.847430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.847410 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.847505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.847485 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.847548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.847536 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.856466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.856446 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft5th\" (UniqueName: \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.914217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.914157 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:53.930091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.930059 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:53.946497 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.946458 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.946616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.946515 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.946616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.946584 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6srw\" (UniqueName: \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.946775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.946624 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.946775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.946664 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.947442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.947415 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.947551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.947424 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.949204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.949180 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.949471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.949448 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.959326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.959299 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6srw\" (UniqueName: \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.982054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.982027 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:53.997509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:53.997479 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:39:54.001422 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:39:54.001399 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad7fb426_0d27_43ce_b11e_e8a917c737e0.slice/crio-dd5493f5656a6c62c8c87e2379e1e02ef5e27c44da9ea4e070f1e38602fbe196 WatchSource:0}: Error finding container dd5493f5656a6c62c8c87e2379e1e02ef5e27c44da9ea4e070f1e38602fbe196: Status 404 returned error can't find the container with id dd5493f5656a6c62c8c87e2379e1e02ef5e27c44da9ea4e070f1e38602fbe196 Oct 31 00:39:54.070391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.070357 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:54.197758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.197667 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:39:54.200840 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:39:54.200800 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2db461e_e1be_425f_a566_007bd63b5cbb.slice/crio-c6d30ee161c638ee7969be4bdaa30e72eff762069f8791b981461a645231d51f WatchSource:0}: Error finding container c6d30ee161c638ee7969be4bdaa30e72eff762069f8791b981461a645231d51f: Status 404 returned error can't find the container with id c6d30ee161c638ee7969be4bdaa30e72eff762069f8791b981461a645231d51f Oct 31 00:39:54.257073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.257041 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:39:54.260435 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:39:54.260403 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb66a96d4_3035_4caa_b2b3_46b02d43d97f.slice/crio-54d386219c4ef6f1caa67b44432a2089e75c1226bee0dc5c8617e039c788b86a WatchSource:0}: Error finding container 54d386219c4ef6f1caa67b44432a2089e75c1226bee0dc5c8617e039c788b86a: Status 404 returned error can't find the container with id 54d386219c4ef6f1caa67b44432a2089e75c1226bee0dc5c8617e039c788b86a Oct 31 00:39:54.879053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.879012 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" event={"ID":"b66a96d4-3035-4caa-b2b3-46b02d43d97f","Type":"ContainerStarted","Data":"54d386219c4ef6f1caa67b44432a2089e75c1226bee0dc5c8617e039c788b86a"} Oct 31 00:39:54.880320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.880294 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" event={"ID":"ad7fb426-0d27-43ce-b11e-e8a917c737e0","Type":"ContainerStarted","Data":"dd5493f5656a6c62c8c87e2379e1e02ef5e27c44da9ea4e070f1e38602fbe196"} Oct 31 00:39:54.881443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:54.881417 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" event={"ID":"f2db461e-e1be-425f-a566-007bd63b5cbb","Type":"ContainerStarted","Data":"c6d30ee161c638ee7969be4bdaa30e72eff762069f8791b981461a645231d51f"} Oct 31 00:39:55.843195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:55.843135 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:55.970026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:55.969983 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:56.734729 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.734684 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:39:56.738630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.738253 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.741827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.741741 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:56.755665 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.755599 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:39:56.819567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.819395 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-grpc\"" Oct 31 00:39:56.819567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.819513 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-index-gateway-http\"" Oct 31 00:39:56.872004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.871955 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.872067 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.872104 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.872341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.872178 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9f7k9\" (UniqueName: \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.872341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.872298 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.875945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.872555 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.879068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.879028 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:39:56.974821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974724 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.974821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974785 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.974821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974813 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.975471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974856 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.975471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974874 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.975471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.974904 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9f7k9\" (UniqueName: \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.976211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.976145 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.976360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.976335 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.979173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.979036 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.979173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.979129 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.981174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.980641 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:56.988576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:56.988502 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9f7k9\" (UniqueName: \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:39:59.296421 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:39:59.296375 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:39:59.907288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.907171 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" event={"ID":"f2db461e-e1be-425f-a566-007bd63b5cbb","Type":"ContainerStarted","Data":"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac"} Oct 31 00:39:59.907288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.907228 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:39:59.908834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.908808 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" event={"ID":"b66a96d4-3035-4caa-b2b3-46b02d43d97f","Type":"ContainerStarted","Data":"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919"} Oct 31 00:39:59.908962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.908860 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:39:59.910249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.910225 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" event={"ID":"ad7fb426-0d27-43ce-b11e-e8a917c737e0","Type":"ContainerStarted","Data":"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a"} Oct 31 00:39:59.910357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.910316 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:39:59.929089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.929031 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" podStartSLOduration=2.232018281 podStartE2EDuration="6.929017376s" totalImagesPullingTime="4.696999095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:59.927687678 +0000 UTC m=+2438.919347262" watchObservedRunningTime="2026-10-31 00:39:59.929017376 +0000 UTC m=+2438.920676959" Oct 31 00:39:59.947795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.947744 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" podStartSLOduration=1.313418951 podStartE2EDuration="6.947730975s" totalImagesPullingTime="5.634312024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:59.946936104 +0000 UTC m=+2438.938595701" watchObservedRunningTime="2026-10-31 00:39:59.947730975 +0000 UTC m=+2438.939390551" Oct 31 00:39:59.966258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:39:59.966214 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" podStartSLOduration=2.316147432 podStartE2EDuration="6.9662016s" totalImagesPullingTime="4.650054168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:59.965904523 +0000 UTC m=+2438.957564108" watchObservedRunningTime="2026-10-31 00:39:59.9662016 +0000 UTC m=+2438.957861183" Oct 31 00:40:01.918430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:01.918399 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:01.925228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:01.925206 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.019109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.019075 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.019682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.019616 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.025061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.025031 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") DevicePath \"csi-914a9fbf68446bc357712d7054dbaf9e951ba6fbcb0a032751039940060a6564\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.240377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.240335 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/8fd5a5ac985486c6416e9f57ebb5ef27608808414a3869903174362f39d872c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.253662 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.253636 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"data-science-usage-logs-lokistack-index-gateway-0\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.453825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.453795 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.732290 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.732261 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:40:02.734894 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:40:02.734866 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5a10009_bc06_4eb3_94d6_7257678e4c68.slice/crio-a85c59e71355f09d04495ffa782dd25e650532540877edea7bc8701e2f847b12 WatchSource:0}: Error finding container a85c59e71355f09d04495ffa782dd25e650532540877edea7bc8701e2f847b12: Status 404 returned error can't find the container with id a85c59e71355f09d04495ffa782dd25e650532540877edea7bc8701e2f847b12 Oct 31 00:40:02.925927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.925884 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"d5a10009-bc06-4eb3-94d6-7257678e4c68","Type":"ContainerStarted","Data":"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b"} Oct 31 00:40:02.926380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.925944 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"d5a10009-bc06-4eb3-94d6-7257678e4c68","Type":"ContainerStarted","Data":"a85c59e71355f09d04495ffa782dd25e650532540877edea7bc8701e2f847b12"} Oct 31 00:40:02.926380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.925986 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:02.947863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:02.947815 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podStartSLOduration=9.947801278 podStartE2EDuration="9.947801278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:02.947075344 +0000 UTC m=+2441.938734949" watchObservedRunningTime="2026-10-31 00:40:02.947801278 +0000 UTC m=+2441.939460862" Oct 31 00:40:10.496755 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:10.496704 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:13.481252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:13.481215 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:20.919748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:20.919700 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:40:20.920249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:20.920190 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:40:20.920331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:20.920317 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:40:23.935619 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:23.935583 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:40:26.703080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:40:26.703039 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:40:29.292846 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:40:29.292810 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:40:59.292395 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:40:59.292356 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:41:22.413417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.413382 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:22.416763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.416737 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.426607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.426586 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:41:22.426767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.426650 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-sm6jl\"" Oct 31 00:41:22.429313 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.429289 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:22.444438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.444411 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:22.506748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.506698 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:41:22.536094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.536065 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.536231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.536108 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhr9\" (UniqueName: \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.536231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.536199 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.587945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.587915 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:22.636897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.636862 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.637093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.636914 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.637167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.637087 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhr9\" (UniqueName: \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.637524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.637497 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.639624 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.639592 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.650083 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.650060 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhr9\" (UniqueName: \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\") pod \"data-science-usage-logs-collector-5dc8ff949d-c8r26\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:22.730251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:22.730170 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:23.006968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.006920 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:23.010150 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:41:23.010123 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdf1444a_efce_495a_b175_5e6be2d81bc8.slice/crio-930ce0c883ab799ba3508474baa74262443d531e1d4e2a0dec631de9c3314060 WatchSource:0}: Error finding container 930ce0c883ab799ba3508474baa74262443d531e1d4e2a0dec631de9c3314060: Status 404 returned error can't find the container with id 930ce0c883ab799ba3508474baa74262443d531e1d4e2a0dec631de9c3314060 Oct 31 00:41:23.012451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.012422 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:41:23.012547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.012481 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:41:23.264029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.263937 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" event={"ID":"bdf1444a-efce-495a-b175-5e6be2d81bc8","Type":"ContainerStarted","Data":"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8"} Oct 31 00:41:23.264029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.263985 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" event={"ID":"bdf1444a-efce-495a-b175-5e6be2d81bc8","Type":"ContainerStarted","Data":"930ce0c883ab799ba3508474baa74262443d531e1d4e2a0dec631de9c3314060"} Oct 31 00:41:23.285277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:23.285221 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" podStartSLOduration=1.285181084 podStartE2EDuration="1.285181084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:23.284756831 +0000 UTC m=+2522.276416416" watchObservedRunningTime="2026-10-31 00:41:23.285181084 +0000 UTC m=+2522.276840667" Oct 31 00:41:26.029918 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:26.029883 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:26.169095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:26.169060 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:26.359203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:26.359109 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:26.481024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:26.480992 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:26.763881 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:26.763812 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.502179 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.502147 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:41:27.505494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.505474 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.516401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.516363 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:41:27.683052 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.683016 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.683244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.683146 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.683307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.683252 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zst5g\" (UniqueName: \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.763825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.763748 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:41:27.764001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.763745 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.767366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.767343 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.781692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.781662 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:41:27.783956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.783936 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.784094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.783985 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zst5g\" (UniqueName: \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.784094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.784039 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.784658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.784628 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.786879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.786858 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.793643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.793621 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-logs-loki-secret\"" Oct 31 00:41:27.794393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.794285 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 31 00:41:27.794393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.794339 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-grpc\"" Oct 31 00:41:27.794393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.794363 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-querier-http\"" Oct 31 00:41:27.794751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.794420 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-ww784\"" Oct 31 00:41:27.795379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.795359 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 31 00:41:27.801808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.801784 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zst5g\" (UniqueName: \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\") pod \"data-science-usage-logs-collector-778d5f8859-fkl9z\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.821212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.821183 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885303 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885364 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885396 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885417 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdlvh\" (UniqueName: \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885438 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.885568 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.885466 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.916430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.916393 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:27.986649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986619 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.986841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986657 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdlvh\" (UniqueName: \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.986841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986683 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.986841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986758 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.986841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986789 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.987066 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.986860 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.987456 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.987428 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.987770 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.987587 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.989554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.989526 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.989554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.989540 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:27.989683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:27.989551 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:28.001308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.001278 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdlvh\" (UniqueName: \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\") pod \"data-science-lokistack-querier-778ddf848f-78ktv\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:28.050487 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.050406 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:28.079272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.079242 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:28.104342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.104308 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:41:28.108064 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:41:28.108042 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ad2097_302f_47d7_94d1_743872331e59.slice/crio-7694534570e52eb5bbeef45ed8e3ce18a3bfdb1310ea8639871b84fcad557787 WatchSource:0}: Error finding container 7694534570e52eb5bbeef45ed8e3ce18a3bfdb1310ea8639871b84fcad557787: Status 404 returned error can't find the container with id 7694534570e52eb5bbeef45ed8e3ce18a3bfdb1310ea8639871b84fcad557787 Oct 31 00:41:28.110925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.110893 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:41:28.111022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.110963 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:41:28.176180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.176150 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:28.290470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.290433 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" event={"ID":"46ad2097-302f-47d7-94d1-743872331e59","Type":"ContainerStarted","Data":"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3"} Oct 31 00:41:28.290635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.290491 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" event={"ID":"46ad2097-302f-47d7-94d1-743872331e59","Type":"ContainerStarted","Data":"7694534570e52eb5bbeef45ed8e3ce18a3bfdb1310ea8639871b84fcad557787"} Oct 31 00:41:28.310485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.310367 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" podStartSLOduration=1.310349976 podStartE2EDuration="1.310349976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:28.309673943 +0000 UTC m=+2527.301333539" watchObservedRunningTime="2026-10-31 00:41:28.310349976 +0000 UTC m=+2527.302009556" Oct 31 00:41:28.341249 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.341214 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:28.341486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.341445 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" podUID="bdf1444a-efce-495a-b175-5e6be2d81bc8" containerName="otc-container" containerID="cri-o://aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8" gracePeriod=30 Oct 31 00:41:28.370105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.370066 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:41:28.396646 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:41:28.396617 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a6a410e_59ec_4b86_bfbe_42ade61c29e8.slice/crio-478e9c83db2f657463f564d8cd801894fc489a978c53a4c952d8e4a372083716 WatchSource:0}: Error finding container 478e9c83db2f657463f564d8cd801894fc489a978c53a4c952d8e4a372083716: Status 404 returned error can't find the container with id 478e9c83db2f657463f564d8cd801894fc489a978c53a4c952d8e4a372083716 Oct 31 00:41:28.576601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.576576 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:28.692037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.691991 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\" (UniqueName: \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\") pod \"bdf1444a-efce-495a-b175-5e6be2d81bc8\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " Oct 31 00:41:28.692221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.692049 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\") pod \"bdf1444a-efce-495a-b175-5e6be2d81bc8\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " Oct 31 00:41:28.692221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.692104 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\") pod \"bdf1444a-efce-495a-b175-5e6be2d81bc8\" (UID: \"bdf1444a-efce-495a-b175-5e6be2d81bc8\") " Oct 31 00:41:28.692552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.692519 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal" pod "bdf1444a-efce-495a-b175-5e6be2d81bc8" (UID: "bdf1444a-efce-495a-b175-5e6be2d81bc8"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:28.694332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.694315 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9" pod "bdf1444a-efce-495a-b175-5e6be2d81bc8" (UID: "bdf1444a-efce-495a-b175-5e6be2d81bc8"). InnerVolumeSpecName "kube-api-access-lbhr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:28.694404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.694345 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs" pod "bdf1444a-efce-495a-b175-5e6be2d81bc8" (UID: "bdf1444a-efce-495a-b175-5e6be2d81bc8"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:28.793786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.793748 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbhr9\" (UniqueName: \"kubernetes.io/projected/bdf1444a-efce-495a-b175-5e6be2d81bc8-kube-api-access-lbhr9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:28.793786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.793788 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/bdf1444a-efce-495a-b175-5e6be2d81bc8-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:28.793984 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:28.793804 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/bdf1444a-efce-495a-b175-5e6be2d81bc8-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:29.294232 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:41:29.294189 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:41:29.296647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.296615 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" event={"ID":"3a6a410e-59ec-4b86-bfbe-42ade61c29e8","Type":"ContainerStarted","Data":"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c"} Oct 31 00:41:29.296798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.296661 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" event={"ID":"3a6a410e-59ec-4b86-bfbe-42ade61c29e8","Type":"ContainerStarted","Data":"478e9c83db2f657463f564d8cd801894fc489a978c53a4c952d8e4a372083716"} Oct 31 00:41:29.296798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.296719 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:29.297853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.297830 5577 generic.go:356] "Generic (PLEG): container finished" podID="bdf1444a-efce-495a-b175-5e6be2d81bc8" containerID="aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8" exitCode=0 Oct 31 00:41:29.297933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.297869 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" Oct 31 00:41:29.297933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.297895 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" event={"ID":"bdf1444a-efce-495a-b175-5e6be2d81bc8","Type":"ContainerDied","Data":"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8"} Oct 31 00:41:29.298018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.297940 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26" event={"ID":"bdf1444a-efce-495a-b175-5e6be2d81bc8","Type":"ContainerDied","Data":"930ce0c883ab799ba3508474baa74262443d531e1d4e2a0dec631de9c3314060"} Oct 31 00:41:29.298018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.297961 5577 scope.go:122] "RemoveContainer" containerID="aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8" Oct 31 00:41:29.306687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.306669 5577 scope.go:122] "RemoveContainer" containerID="aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8" Oct 31 00:41:29.307001 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:41:29.306975 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8\": container with ID starting with aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8 not found: ID does not exist" containerID="aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8" Oct 31 00:41:29.307064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.307014 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8"} err="failed to get container status \"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8\": rpc error: code = NotFound desc = could not find container \"aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8\": container with ID starting with aea74c58707e0eddf43b41f210917d5840adbf144137782d04b1e89013a3f0d8 not found: ID does not exist" Oct 31 00:41:29.318233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.318189 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" podStartSLOduration=2.318176799 podStartE2EDuration="2.318176799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:29.316965745 +0000 UTC m=+2528.308625344" watchObservedRunningTime="2026-10-31 00:41:29.318176799 +0000 UTC m=+2528.309836380" Oct 31 00:41:29.331958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.331924 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:29.336283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:29.336256 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5dc8ff949d-c8r26"] Oct 31 00:41:30.031064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.031028 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.152241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.152209 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:30.836968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.836935 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:41:30.837289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.837248 5577 cpu_manager.go:414] "containerMap: removing container" podUID="bdf1444a-efce-495a-b175-5e6be2d81bc8" containerName="otc-container" Oct 31 00:41:30.837333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.837290 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bdf1444a-efce-495a-b175-5e6be2d81bc8" containerName="otc-container" Oct 31 00:41:30.837368 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.837362 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="bdf1444a-efce-495a-b175-5e6be2d81bc8" containerName="otc-container" Oct 31 00:41:30.841113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.841095 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:30.851606 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.851584 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:41:30.874115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.874089 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\"" Oct 31 00:41:30.874256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.874154 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\"" Oct 31 00:41:30.912940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:30.912906 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.014362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014330 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.014566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014387 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.014566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014483 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.014566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014512 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.014566 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014555 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.014805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.014665 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9h7j\" (UniqueName: \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.040210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.040173 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.115771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115676 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9h7j\" (UniqueName: \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.115771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115755 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.115983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115789 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.115983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115822 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.115983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115849 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.115983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.115903 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.116588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.116558 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.116870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.116615 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.118580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.118557 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.119026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.119008 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.119181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.119163 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.129060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.129041 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9h7j\" (UniqueName: \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:31.129237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.129218 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bdf1444a-efce-495a-b175-5e6be2d81bc8" path="/var/lib/kubelet/pods/bdf1444a-efce-495a-b175-5e6be2d81bc8/volumes" Oct 31 00:41:31.175647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.175612 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.723732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.723673 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q6vc7"] Oct 31 00:41:31.728209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.728189 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.741384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.741359 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q6vc7"] Oct 31 00:41:31.775512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.775480 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:31.775512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.775505 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:31.775702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.775564 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:31.783102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.783069 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.791925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.791896 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q6vc7"] Oct 31 00:41:31.792540 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:41:31.792509 5577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[config config-openshift-service-cacrt data-science-cluster-log-forwarder-collector-token datadir entrypoint kube-api-access-klc87 metrics sa-token tmp trusted-ca varlogpods], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" podUID="b7d81c71-6f23-47a8-9592-0332118a34db" Oct 31 00:41:31.813518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.813495 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:31.815908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.815890 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:31.823322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823294 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823352 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823486 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823520 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823543 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823560 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823652 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823640 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823720 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823688 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823799 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823845 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc87\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.823943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.823917 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.907982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.907939 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:41:31.925113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925081 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925124 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925158 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925183 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925192 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925200 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925299 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925333 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925383 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925414 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925466 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925492 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc87\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925966 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925898 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.925966 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.925896 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.926339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.926319 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.926545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.926517 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.926865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.926840 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.928020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.928002 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.928119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.928058 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.928182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.928165 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.937899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.937879 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:31.938041 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:31.938024 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc87\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\") pod \"data-science-cluster-log-forwarder-q6vc7\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:32.312176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.312144 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:32.320351 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.320327 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:32.431074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431038 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431077 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431112 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431149 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431178 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431212 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431262 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431289 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431320 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431389 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431423 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431467 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\") pod \"b7d81c71-6f23-47a8-9592-0332118a34db\" (UID: \"b7d81c71-6f23-47a8-9592-0332118a34db\") " Oct 31 00:41:32.431699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431531 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.431699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431567 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:41:32.432011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431801 5577 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-varlogpods\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.432011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431823 5577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-trusted-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.432011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431841 5577 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/b7d81c71-6f23-47a8-9592-0332118a34db-datadir\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.432011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.431947 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.432189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.432009 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.432335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.432313 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:41:32.433748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.433699 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:41:32.433991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.433968 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.434088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.434017 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.434088 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.434045 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "kube-api-access-klc87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:41:32.434304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.434278 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token" pod "b7d81c71-6f23-47a8-9592-0332118a34db" (UID: "b7d81c71-6f23-47a8-9592-0332118a34db"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:41:32.532871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532839 5577 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-metrics\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.532871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532865 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.532871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532875 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/b7d81c71-6f23-47a8-9592-0332118a34db-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.533114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532885 5577 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b7d81c71-6f23-47a8-9592-0332118a34db-tmp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.533114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532894 5577 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-entrypoint\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.533114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532907 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klc87\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-kube-api-access-klc87\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.533114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532921 5577 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/b7d81c71-6f23-47a8-9592-0332118a34db-config-openshift-service-cacrt\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:32.533114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:32.532929 5577 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/b7d81c71-6f23-47a8-9592-0332118a34db-sa-token\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:41:33.316492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.316463 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-q6vc7" Oct 31 00:41:33.351468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.351438 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q6vc7"] Oct 31 00:41:33.356072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.356044 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-q6vc7"] Oct 31 00:41:33.375124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.375096 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:41:33.379467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.379443 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.383358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.383334 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 31 00:41:33.383525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.383425 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-fkhp6\"" Oct 31 00:41:33.383525 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.383509 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 31 00:41:33.384207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.384154 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 31 00:41:33.386308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.386290 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 31 00:41:33.389527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.389504 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:41:33.542939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.542898 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.542939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.542942 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.542971 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543076 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543135 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543180 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543237 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543291 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7sc6\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543320 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543337 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.543544 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.543445 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644105 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644151 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644185 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644227 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644265 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644287 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644314 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644323 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644393 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644426 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7sc6\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644456 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644479 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644567 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.644958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644917 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.645051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.644964 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.645458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.645436 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.646000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.645978 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.647026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.647003 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.647026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.647015 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.647215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.647192 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.660579 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.660552 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.660780 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.660763 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7sc6\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\") pod \"data-science-cluster-log-forwarder-j6jjb\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.693869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.693840 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:41:33.745405 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.745372 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:33.755647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.752413 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:33.846169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.846129 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:33.846335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.846235 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:33.849794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.849777 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") DevicePath \"csi-9219bab33e94be57739cf2870bc6427a58c9ff9febc7933fe4cf49c24807ed45\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:33.970529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:33.970445 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:41:33.973289 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:41:33.973252 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf72f403b_78fe_467c_afed_60dcfaa04e74.slice/crio-b742b3967bfbf3498ab9201fe0d07220b9ce988dedce02b99806c57e4b6186f2 WatchSource:0}: Error finding container b742b3967bfbf3498ab9201fe0d07220b9ce988dedce02b99806c57e4b6186f2: Status 404 returned error can't find the container with id b742b3967bfbf3498ab9201fe0d07220b9ce988dedce02b99806c57e4b6186f2 Oct 31 00:41:34.069473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:34.069446 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/d4a91a7554a4fee5546387ff6a57e9af5a0265dbf8801fa265df6ea6953d9286/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:34.083321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:34.083294 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"data-science-lokistack-compactor-0\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:34.152415 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:34.152382 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:34.322202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:34.322169 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" event={"ID":"f72f403b-78fe-467c-afed-60dcfaa04e74","Type":"ContainerStarted","Data":"b742b3967bfbf3498ab9201fe0d07220b9ce988dedce02b99806c57e4b6186f2"} Oct 31 00:41:34.427694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:34.427658 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:41:34.432131 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:41:34.432106 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb99e0298_a2be_48da_a3f4_76d963a42bb7.slice/crio-5c55ebce45f13154a358610ab6a102d941ebf0abe5f8d853f8853ecb052d0493 WatchSource:0}: Error finding container 5c55ebce45f13154a358610ab6a102d941ebf0abe5f8d853f8853ecb052d0493: Status 404 returned error can't find the container with id 5c55ebce45f13154a358610ab6a102d941ebf0abe5f8d853f8853ecb052d0493 Oct 31 00:41:35.129853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:35.129820 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d81c71-6f23-47a8-9592-0332118a34db" path="/var/lib/kubelet/pods/b7d81c71-6f23-47a8-9592-0332118a34db/volumes" Oct 31 00:41:35.328506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:35.328469 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"b99e0298-a2be-48da-a3f4-76d963a42bb7","Type":"ContainerStarted","Data":"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de"} Oct 31 00:41:35.328506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:35.328513 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"b99e0298-a2be-48da-a3f4-76d963a42bb7","Type":"ContainerStarted","Data":"5c55ebce45f13154a358610ab6a102d941ebf0abe5f8d853f8853ecb052d0493"} Oct 31 00:41:35.329063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:35.328611 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:35.356094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:35.350561 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=8.35053208 podStartE2EDuration="8.35053208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:35.348541855 +0000 UTC m=+2534.340201449" watchObservedRunningTime="2026-10-31 00:41:35.35053208 +0000 UTC m=+2534.342191662" Oct 31 00:41:39.348817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:39.348780 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" event={"ID":"f72f403b-78fe-467c-afed-60dcfaa04e74","Type":"ContainerStarted","Data":"9ed9b9b4919acffc15b2c236e07599d4936ba6deb451c694225bd2464182f6c9"} Oct 31 00:41:39.373019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:39.372964 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" podStartSLOduration=1.472477918 podStartE2EDuration="6.372949038s" totalImagesPullingTime="4.90047112s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:41:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:39.371632438 +0000 UTC m=+2538.363292054" watchObservedRunningTime="2026-10-31 00:41:39.372949038 +0000 UTC m=+2538.364608622" Oct 31 00:41:50.309483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:50.309448 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:41:56.340822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:41:56.340785 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:41:59.292866 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:41:59.292827 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:42:26.811957 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:26.811895 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:29.292351 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:29.292314 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:42:45.320369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.320334 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:45.323671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.323655 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.334562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.334531 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:45.403388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.403165 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.403808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.403779 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6nrf\" (UniqueName: \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.404004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.403967 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.436808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.436762 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.488915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.488886 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:42:45.489098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.489069 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerName="loki-compactor" containerID="cri-o://95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de" gracePeriod=30 Oct 31 00:42:45.505764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.505374 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.505764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.505432 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6nrf\" (UniqueName: \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.505764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.505479 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.506231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.506207 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.508644 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.508619 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.526273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.526225 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6nrf\" (UniqueName: \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\") pod \"data-science-usage-logs-collector-9f68bf844-qwvbq\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.585769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.585637 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.594493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.594465 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:42:45.594767 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.594743 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" podUID="f72f403b-78fe-467c-afed-60dcfaa04e74" containerName="collector" containerID="cri-o://9ed9b9b4919acffc15b2c236e07599d4936ba6deb451c694225bd2464182f6c9" gracePeriod=10 Oct 31 00:42:45.601664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.601629 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:42:45.602067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.601863 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" podUID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" containerName="loki-querier" containerID="cri-o://7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c" gracePeriod=30 Oct 31 00:42:45.636891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.636859 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:45.717457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.717333 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:45.916293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.916261 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:45.919379 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:42:45.919347 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e7726a_46de_47aa_aacf_6bba1e622d02.slice/crio-a8fe18881b99e7a5daaedf4cde2c15f2c2fe4f9389f2418aa39665eb5e6a4a75 WatchSource:0}: Error finding container a8fe18881b99e7a5daaedf4cde2c15f2c2fe4f9389f2418aa39665eb5e6a4a75: Status 404 returned error can't find the container with id a8fe18881b99e7a5daaedf4cde2c15f2c2fe4f9389f2418aa39665eb5e6a4a75 Oct 31 00:42:45.921700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.921664 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:42:45.921821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:45.921751 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:42:46.048982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.048956 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:42:46.112044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.111944 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112017 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112090 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112128 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\" (UniqueName: \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112224 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112266 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\") pod \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\" (UID: \"3a6a410e-59ec-4b86-bfbe-42ade61c29e8\") " Oct 31 00:42:46.112540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112358 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.112637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.112620 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.113208 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.113159 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.114794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.114742 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.114911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.114887 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "kube-api-access-wdlvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.115242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.115215 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "data-science-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.116233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.116212 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http" pod "3a6a410e-59ec-4b86-bfbe-42ade61c29e8" (UID: "3a6a410e-59ec-4b86-bfbe-42ade61c29e8"). InnerVolumeSpecName "data-science-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.213629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.213597 5577 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-test-logs-loki-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.213629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.213626 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.213629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.213637 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdlvh\" (UniqueName: \"kubernetes.io/projected/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-kube-api-access-wdlvh\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.213908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.213649 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-data-science-lokistack-querier-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.213908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.213658 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3a6a410e-59ec-4b86-bfbe-42ade61c29e8-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.339390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.339348 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerName="loki-compactor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:46.639189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.639167 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:42:46.650114 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.650087 5577 generic.go:356] "Generic (PLEG): container finished" podID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" containerID="7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c" exitCode=0 Oct 31 00:42:46.650257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.650122 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" Oct 31 00:42:46.650257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.650138 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" event={"ID":"3a6a410e-59ec-4b86-bfbe-42ade61c29e8","Type":"ContainerDied","Data":"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c"} Oct 31 00:42:46.650257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.650180 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv" event={"ID":"3a6a410e-59ec-4b86-bfbe-42ade61c29e8","Type":"ContainerDied","Data":"478e9c83db2f657463f564d8cd801894fc489a978c53a4c952d8e4a372083716"} Oct 31 00:42:46.650257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.650204 5577 scope.go:122] "RemoveContainer" containerID="7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c" Oct 31 00:42:46.651378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.651356 5577 generic.go:356] "Generic (PLEG): container finished" podID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerID="95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de" exitCode=0 Oct 31 00:42:46.651543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.651399 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 31 00:42:46.651543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.651453 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"b99e0298-a2be-48da-a3f4-76d963a42bb7","Type":"ContainerDied","Data":"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de"} Oct 31 00:42:46.651543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.651488 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"b99e0298-a2be-48da-a3f4-76d963a42bb7","Type":"ContainerDied","Data":"5c55ebce45f13154a358610ab6a102d941ebf0abe5f8d853f8853ecb052d0493"} Oct 31 00:42:46.653293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.653267 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" event={"ID":"29e7726a-46de-47aa-aacf-6bba1e622d02","Type":"ContainerStarted","Data":"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d"} Oct 31 00:42:46.653386 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.653308 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" event={"ID":"29e7726a-46de-47aa-aacf-6bba1e622d02","Type":"ContainerStarted","Data":"a8fe18881b99e7a5daaedf4cde2c15f2c2fe4f9389f2418aa39665eb5e6a4a75"} Oct 31 00:42:46.660122 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.659793 5577 scope.go:122] "RemoveContainer" containerID="7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c" Oct 31 00:42:46.660856 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:46.660538 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c\": container with ID starting with 7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c not found: ID does not exist" containerID="7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c" Oct 31 00:42:46.660856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.660572 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c"} err="failed to get container status \"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c\": rpc error: code = NotFound desc = could not find container \"7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c\": container with ID starting with 7bf8ac257055463e10afd6a465da290b5b5764ac59d2c71260e43707415d584c not found: ID does not exist" Oct 31 00:42:46.660856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.660595 5577 scope.go:122] "RemoveContainer" containerID="95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de" Oct 31 00:42:46.670032 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.670013 5577 scope.go:122] "RemoveContainer" containerID="95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de" Oct 31 00:42:46.670309 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:46.670291 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de\": container with ID starting with 95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de not found: ID does not exist" containerID="95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de" Oct 31 00:42:46.670363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.670316 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de"} err="failed to get container status \"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de\": rpc error: code = NotFound desc = could not find container \"95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de\": container with ID starting with 95b724b2ff7936c4c98d0211a801f9013ac0d194c6973845bb1b8dc2fb3536de not found: ID does not exist" Oct 31 00:42:46.678314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.678269 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" podStartSLOduration=1.678253213 podStartE2EDuration="1.678253213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:46.67769729 +0000 UTC m=+2605.669356879" watchObservedRunningTime="2026-10-31 00:42:46.678253213 +0000 UTC m=+2605.669912801" Oct 31 00:42:46.692504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.692479 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:42:46.696337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.696318 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-querier-778ddf848f-78ktv"] Oct 31 00:42:46.707465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.707441 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:42:46.707636 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.707617 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" podUID="46ad2097-302f-47d7-94d1-743872331e59" containerName="otc-container" containerID="cri-o://909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3" gracePeriod=30 Oct 31 00:42:46.718413 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718390 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718436 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\" (UniqueName: \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718529 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718577 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718663 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718608 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718753 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.718854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.718806 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\") pod \"b99e0298-a2be-48da-a3f4-76d963a42bb7\" (UID: \"b99e0298-a2be-48da-a3f4-76d963a42bb7\") " Oct 31 00:42:46.721941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.721911 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.723640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.722266 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:46.728364 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.724222 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "kube-api-access-d9h7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:46.728364 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.725941 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "pvc-2efabd0a-4bd4-4509-8865-4a5675f61765". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:46.737897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.737681 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.739482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.739454 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.739542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.739464 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret" pod "b99e0298-a2be-48da-a3f4-76d963a42bb7" (UID: "b99e0298-a2be-48da-a3f4-76d963a42bb7"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:46.822123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822090 5577 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-test-logs-loki-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.822123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822120 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.822123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822132 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.822419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822171 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:42:46.822419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822188 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.822419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822202 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/b99e0298-a2be-48da-a3f4-76d963a42bb7-data-science-lokistack-compactor-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.822419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.822216 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9h7j\" (UniqueName: \"kubernetes.io/projected/b99e0298-a2be-48da-a3f4-76d963a42bb7-kube-api-access-d9h7j\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:46.833346 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.833326 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-2efabd0a-4bd4-4509-8865-4a5675f61765" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:42:46.923013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.922943 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-2efabd0a-4bd4-4509-8865-4a5675f61765\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0bed796cabab4394a\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-9219bab33e94be57739cf2870bc6427a58c9ff9febc7933fe4cf49c24807ed45\"" Oct 31 00:42:46.967082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.967051 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:42:46.975000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.974973 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:42:46.978629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:46.978590 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 31 00:42:47.023650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.023615 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\") pod \"46ad2097-302f-47d7-94d1-743872331e59\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " Oct 31 00:42:47.023827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.023679 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\" (UniqueName: \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\") pod \"46ad2097-302f-47d7-94d1-743872331e59\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " Oct 31 00:42:47.023827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.023795 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\" (UniqueName: \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\") pod \"46ad2097-302f-47d7-94d1-743872331e59\" (UID: \"46ad2097-302f-47d7-94d1-743872331e59\") " Oct 31 00:42:47.024045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.024021 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal" pod "46ad2097-302f-47d7-94d1-743872331e59" (UID: "46ad2097-302f-47d7-94d1-743872331e59"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:47.024140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.024125 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/46ad2097-302f-47d7-94d1-743872331e59-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.025863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.025841 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs" pod "46ad2097-302f-47d7-94d1-743872331e59" (UID: "46ad2097-302f-47d7-94d1-743872331e59"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:47.025985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.025960 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g" pod "46ad2097-302f-47d7-94d1-743872331e59" (UID: "46ad2097-302f-47d7-94d1-743872331e59"). InnerVolumeSpecName "kube-api-access-zst5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:47.125401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.125366 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/46ad2097-302f-47d7-94d1-743872331e59-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.125401 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.125393 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zst5g\" (UniqueName: \"kubernetes.io/projected/46ad2097-302f-47d7-94d1-743872331e59-kube-api-access-zst5g\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:47.129940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.129912 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" path="/var/lib/kubelet/pods/3a6a410e-59ec-4b86-bfbe-42ade61c29e8/volumes" Oct 31 00:42:47.130342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.130330 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" path="/var/lib/kubelet/pods/b99e0298-a2be-48da-a3f4-76d963a42bb7/volumes" Oct 31 00:42:47.660164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.660134 5577 generic.go:356] "Generic (PLEG): container finished" podID="46ad2097-302f-47d7-94d1-743872331e59" containerID="909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3" exitCode=0 Oct 31 00:42:47.660601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.660199 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" Oct 31 00:42:47.660601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.660231 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" event={"ID":"46ad2097-302f-47d7-94d1-743872331e59","Type":"ContainerDied","Data":"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3"} Oct 31 00:42:47.660601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.660291 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z" event={"ID":"46ad2097-302f-47d7-94d1-743872331e59","Type":"ContainerDied","Data":"7694534570e52eb5bbeef45ed8e3ce18a3bfdb1310ea8639871b84fcad557787"} Oct 31 00:42:47.660601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.660322 5577 scope.go:122] "RemoveContainer" containerID="909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3" Oct 31 00:42:47.671558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.671538 5577 scope.go:122] "RemoveContainer" containerID="909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3" Oct 31 00:42:47.671859 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:47.671837 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3\": container with ID starting with 909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3 not found: ID does not exist" containerID="909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3" Oct 31 00:42:47.671955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.671862 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3"} err="failed to get container status \"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3\": rpc error: code = NotFound desc = could not find container \"909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3\": container with ID starting with 909286030e22a004ed9f5626094137f0bfcc97ca89933305e3b492976c9adea3 not found: ID does not exist" Oct 31 00:42:47.678291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.678271 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:42:47.682630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:47.682607 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-778d5f8859-fkl9z"] Oct 31 00:42:49.129382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:49.129341 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ad2097-302f-47d7-94d1-743872331e59" path="/var/lib/kubelet/pods/46ad2097-302f-47d7-94d1-743872331e59/volumes" Oct 31 00:42:53.067265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.067232 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:42:53.067743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.067435 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" podUID="d5a10009-bc06-4eb3-94d6-7257678e4c68" containerName="loki-index-gateway" containerID="cri-o://543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b" gracePeriod=30 Oct 31 00:42:53.189257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.189217 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:53.317837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.317745 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:53.332881 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.332857 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.482291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482262 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482314 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482341 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\" (UniqueName: \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482373 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482397 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482419 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482456 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\") pod \"d5a10009-bc06-4eb3-94d6-7257678e4c68\" (UID: \"d5a10009-bc06-4eb3-94d6-7257678e4c68\") " Oct 31 00:42:53.482847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482759 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:53.482904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482858 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.483019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.482901 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:53.484948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.484910 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "kube-api-access-9f7k9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:53.485189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.485152 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:53.485189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.485176 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "data-science-usage-logs-lokistack-index-gateway-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:53.485377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.485359 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:53.485677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.485656 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42" pod "d5a10009-bc06-4eb3-94d6-7257678e4c68" (UID: "d5a10009-bc06-4eb3-94d6-7257678e4c68"). InnerVolumeSpecName "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584035 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584064 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9f7k9\" (UniqueName: \"kubernetes.io/projected/d5a10009-bc06-4eb3-94d6-7257678e4c68-kube-api-access-9f7k9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584075 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-grpc\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584085 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5a10009-bc06-4eb3-94d6-7257678e4c68-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584095 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-index-gateway-http\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-data-science-usage-logs-lokistack-index-gateway-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.584119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.584104 5577 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d5a10009-bc06-4eb3-94d6-7257678e4c68-test-usage-logs-loki-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:53.595335 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.595314 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-d40c9d92-b1bc-4286-b370-a2b061c99081" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:42:53.685009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.684976 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-d40c9d92-b1bc-4286-b370-a2b061c99081\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0605dd9a241de2a42\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-914a9fbf68446bc357712d7054dbaf9e951ba6fbcb0a032751039940060a6564\"" Oct 31 00:42:53.690517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.690485 5577 generic.go:356] "Generic (PLEG): container finished" podID="d5a10009-bc06-4eb3-94d6-7257678e4c68" containerID="543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b" exitCode=0 Oct 31 00:42:53.690667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.690533 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" Oct 31 00:42:53.690667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.690577 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"d5a10009-bc06-4eb3-94d6-7257678e4c68","Type":"ContainerDied","Data":"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b"} Oct 31 00:42:53.690667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.690617 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0" event={"ID":"d5a10009-bc06-4eb3-94d6-7257678e4c68","Type":"ContainerDied","Data":"a85c59e71355f09d04495ffa782dd25e650532540877edea7bc8701e2f847b12"} Oct 31 00:42:53.690667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.690634 5577 scope.go:122] "RemoveContainer" containerID="543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b" Oct 31 00:42:53.700501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.700484 5577 scope.go:122] "RemoveContainer" containerID="543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b" Oct 31 00:42:53.700811 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:53.700787 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b\": container with ID starting with 543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b not found: ID does not exist" containerID="543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b" Oct 31 00:42:53.700889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.700821 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b"} err="failed to get container status \"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b\": rpc error: code = NotFound desc = could not find container \"543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b\": container with ID starting with 543405928a96170cbec0d418076531a1b3720bd1665386b56d2a297d8b236c4b not found: ID does not exist" Oct 31 00:42:53.711842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.711820 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:42:53.717474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.717452 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-index-gateway-0"] Oct 31 00:42:53.818825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.818791 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:53.819092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.818996 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" podUID="29e7726a-46de-47aa-aacf-6bba1e622d02" containerName="otc-container" containerID="cri-o://2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d" gracePeriod=30 Oct 31 00:42:53.894765 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:53.894675 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:54.030960 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.030870 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:42:54.076258 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.076226 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:54.187861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.187767 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\" (UniqueName: \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\") pod \"29e7726a-46de-47aa-aacf-6bba1e622d02\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " Oct 31 00:42:54.187861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.187853 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\") pod \"29e7726a-46de-47aa-aacf-6bba1e622d02\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " Oct 31 00:42:54.188129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.187953 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\" (UniqueName: \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\") pod \"29e7726a-46de-47aa-aacf-6bba1e622d02\" (UID: \"29e7726a-46de-47aa-aacf-6bba1e622d02\") " Oct 31 00:42:54.188199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.188147 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal" pod "29e7726a-46de-47aa-aacf-6bba1e622d02" (UID: "29e7726a-46de-47aa-aacf-6bba1e622d02"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:54.188251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.188224 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/29e7726a-46de-47aa-aacf-6bba1e622d02-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.190173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.190149 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs" pod "29e7726a-46de-47aa-aacf-6bba1e622d02" (UID: "29e7726a-46de-47aa-aacf-6bba1e622d02"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:54.190283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.190219 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf" pod "29e7726a-46de-47aa-aacf-6bba1e622d02" (UID: "29e7726a-46de-47aa-aacf-6bba1e622d02"). InnerVolumeSpecName "kube-api-access-g6nrf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:54.288880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.288847 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/29e7726a-46de-47aa-aacf-6bba1e622d02-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.288880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.288881 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6nrf\" (UniqueName: \"kubernetes.io/projected/29e7726a-46de-47aa-aacf-6bba1e622d02-kube-api-access-g6nrf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:54.410002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.409967 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:42:54.410275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410262 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerName="loki-compactor" Oct 31 00:42:54.410322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410275 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerName="loki-compactor" Oct 31 00:42:54.410322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410291 5577 cpu_manager.go:414] "containerMap: removing container" podUID="46ad2097-302f-47d7-94d1-743872331e59" containerName="otc-container" Oct 31 00:42:54.410322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410296 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ad2097-302f-47d7-94d1-743872331e59" containerName="otc-container" Oct 31 00:42:54.410322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410309 5577 cpu_manager.go:414] "containerMap: removing container" podUID="29e7726a-46de-47aa-aacf-6bba1e622d02" containerName="otc-container" Oct 31 00:42:54.410322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410314 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e7726a-46de-47aa-aacf-6bba1e622d02" containerName="otc-container" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410325 5577 cpu_manager.go:414] "containerMap: removing container" podUID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" containerName="loki-querier" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410329 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" containerName="loki-querier" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410339 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d5a10009-bc06-4eb3-94d6-7257678e4c68" containerName="loki-index-gateway" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410344 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a10009-bc06-4eb3-94d6-7257678e4c68" containerName="loki-index-gateway" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410389 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e7726a-46de-47aa-aacf-6bba1e622d02" containerName="otc-container" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410396 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a6a410e-59ec-4b86-bfbe-42ade61c29e8" containerName="loki-querier" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410403 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a10009-bc06-4eb3-94d6-7257678e4c68" containerName="loki-index-gateway" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410410 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b99e0298-a2be-48da-a3f4-76d963a42bb7" containerName="loki-compactor" Oct 31 00:42:54.410560 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.410416 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ad2097-302f-47d7-94d1-743872331e59" containerName="otc-container" Oct 31 00:42:54.413277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.413261 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.423299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.423277 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:42:54.445747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.445651 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 31 00:42:54.445747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.445680 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 31 00:42:54.445902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.445758 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 31 00:42:54.490788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.490760 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.490897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.490799 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.490897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.490822 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.491004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.490967 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.491067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.491017 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfrk\" (UniqueName: \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.491139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.491119 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.591970 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592030 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592055 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592078 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592180 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592103 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592249 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfrk\" (UniqueName: \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592826 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.592875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.592866 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.594942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.594912 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.594942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.594920 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.595202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.595184 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.604298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.604275 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfrk\" (UniqueName: \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:42:54.696118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.696031 5577 generic.go:356] "Generic (PLEG): container finished" podID="29e7726a-46de-47aa-aacf-6bba1e622d02" containerID="2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d" exitCode=0 Oct 31 00:42:54.696118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.696072 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" Oct 31 00:42:54.696333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.696118 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" event={"ID":"29e7726a-46de-47aa-aacf-6bba1e622d02","Type":"ContainerDied","Data":"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d"} Oct 31 00:42:54.696333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.696169 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq" event={"ID":"29e7726a-46de-47aa-aacf-6bba1e622d02","Type":"ContainerDied","Data":"a8fe18881b99e7a5daaedf4cde2c15f2c2fe4f9389f2418aa39665eb5e6a4a75"} Oct 31 00:42:54.696333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.696198 5577 scope.go:122] "RemoveContainer" containerID="2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d" Oct 31 00:42:54.704998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.704980 5577 scope.go:122] "RemoveContainer" containerID="2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d" Oct 31 00:42:54.705260 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:54.705244 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d\": container with ID starting with 2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d not found: ID does not exist" containerID="2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d" Oct 31 00:42:54.705316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.705268 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d"} err="failed to get container status \"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d\": rpc error: code = NotFound desc = could not find container \"2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d\": container with ID starting with 2abedae4483a5b5a276c4816bbae620b46ddac9fb5006aa8b5f0cbd31f98461d not found: ID does not exist" Oct 31 00:42:54.717564 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.717538 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:54.721733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:54.721696 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-9f68bf844-qwvbq"] Oct 31 00:42:55.132467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.132428 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e7726a-46de-47aa-aacf-6bba1e622d02" path="/var/lib/kubelet/pods/29e7726a-46de-47aa-aacf-6bba1e622d02/volumes" Oct 31 00:42:55.133104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.133082 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a10009-bc06-4eb3-94d6-7257678e4c68" path="/var/lib/kubelet/pods/d5a10009-bc06-4eb3-94d6-7257678e4c68/volumes" Oct 31 00:42:55.704882 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.704856 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-logging_data-science-cluster-log-forwarder-j6jjb_f72f403b-78fe-467c-afed-60dcfaa04e74/collector/0.log" Oct 31 00:42:55.705016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.704899 5577 generic.go:356] "Generic (PLEG): container finished" podID="f72f403b-78fe-467c-afed-60dcfaa04e74" containerID="9ed9b9b4919acffc15b2c236e07599d4936ba6deb451c694225bd2464182f6c9" exitCode=137 Oct 31 00:42:55.705016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.704970 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" event={"ID":"f72f403b-78fe-467c-afed-60dcfaa04e74","Type":"ContainerDied","Data":"9ed9b9b4919acffc15b2c236e07599d4936ba6deb451c694225bd2464182f6c9"} Oct 31 00:42:55.768417 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.768392 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-logging_data-science-cluster-log-forwarder-j6jjb_f72f403b-78fe-467c-afed-60dcfaa04e74/collector/0.log" Oct 31 00:42:55.770744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.770702 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:42:55.903301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903211 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903250 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903269 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903299 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903327 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903350 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903386 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903420 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903461 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903515 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903582 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903619 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.903641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903644 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\") pod \"f72f403b-78fe-467c-afed-60dcfaa04e74\" (UID: \"f72f403b-78fe-467c-afed-60dcfaa04e74\") " Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903848 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903854 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.903879 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.904029 5577 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-datadir\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.904049 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.904063 5577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-trusted-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.904080 5577 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/f72f403b-78fe-467c-afed-60dcfaa04e74-varlogpods\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.904155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.904097 5577 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-config-openshift-service-cacrt\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:55.906228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.906200 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "kube-api-access-g7sc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:55.906329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.906199 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:55.906329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.906265 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:55.906409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.906378 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:42:55.906477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.906463 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:55.909671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:55.909649 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint" pod "f72f403b-78fe-467c-afed-60dcfaa04e74" (UID: "f72f403b-78fe-467c-afed-60dcfaa04e74"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:56.004903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004870 5577 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-metrics\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.004903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004898 5577 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f72f403b-78fe-467c-afed-60dcfaa04e74-tmp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.004903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004907 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7sc6\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-kube-api-access-g7sc6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.005136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004918 5577 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/f72f403b-78fe-467c-afed-60dcfaa04e74-sa-token\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.005136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004926 5577 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/f72f403b-78fe-467c-afed-60dcfaa04e74-entrypoint\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.005136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.004935 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/f72f403b-78fe-467c-afed-60dcfaa04e74-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:56.710514 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.710482 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-logging_data-science-cluster-log-forwarder-j6jjb_f72f403b-78fe-467c-afed-60dcfaa04e74/collector/0.log" Oct 31 00:42:56.711028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.710586 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" Oct 31 00:42:56.711028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.710594 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-j6jjb" event={"ID":"f72f403b-78fe-467c-afed-60dcfaa04e74","Type":"ContainerDied","Data":"b742b3967bfbf3498ab9201fe0d07220b9ce988dedce02b99806c57e4b6186f2"} Oct 31 00:42:56.711028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.710642 5577 scope.go:122] "RemoveContainer" containerID="9ed9b9b4919acffc15b2c236e07599d4936ba6deb451c694225bd2464182f6c9" Oct 31 00:42:56.732330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.732305 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:42:56.736672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:56.736643 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-j6jjb"] Oct 31 00:42:57.129980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:42:57.129949 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f72f403b-78fe-467c-afed-60dcfaa04e74" path="/var/lib/kubelet/pods/f72f403b-78fe-467c-afed-60dcfaa04e74/volumes" Oct 31 00:42:59.291950 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:42:59.291869 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:43:06.597865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.597830 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.604749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.604724 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.698833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.698790 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.698990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.698908 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.702285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.702265 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") DevicePath \"csi-142ae4646c2c18dcb6978f1388429458a476710589bcb026ecd45ad140ae0094\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.793239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.793202 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ce8a44ea6fd624fabb855523df239646d3dcc3258d2c5f84025b15690a2664c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:06.807400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:06.807374 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:07.024001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.023958 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:07.299570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.299499 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:43:07.303205 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:43:07.303174 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a194581_246e_4476_b18b_baa47d436613.slice/crio-4174fdc1c85a251a6aa91de912c1078d6bfd4a0e48b2a53a5733643b55ab7c56 WatchSource:0}: Error finding container 4174fdc1c85a251a6aa91de912c1078d6bfd4a0e48b2a53a5733643b55ab7c56: Status 404 returned error can't find the container with id 4174fdc1c85a251a6aa91de912c1078d6bfd4a0e48b2a53a5733643b55ab7c56 Oct 31 00:43:07.761406 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.761371 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"2a194581-246e-4476-b18b-baa47d436613","Type":"ContainerStarted","Data":"b131dd2c26dbdd912f0e64d570fc4319fda04061182841c38737261d7978e4ab"} Oct 31 00:43:07.761828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.761418 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"2a194581-246e-4476-b18b-baa47d436613","Type":"ContainerStarted","Data":"4174fdc1c85a251a6aa91de912c1078d6bfd4a0e48b2a53a5733643b55ab7c56"} Oct 31 00:43:07.761828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.761452 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:07.783366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:07.783326 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=13.783313095 podStartE2EDuration="13.783313095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:42:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:07.781696118 +0000 UTC m=+2626.773355702" watchObservedRunningTime="2026-10-31 00:43:07.783313095 +0000 UTC m=+2626.774972679" Oct 31 00:43:15.380928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:15.380889 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:43:15.381434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:15.381113 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" podUID="f2db461e-e1be-425f-a566-007bd63b5cbb" containerName="loki-querier" containerID="cri-o://8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac" gracePeriod=30 Oct 31 00:43:15.953846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:15.953819 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:43:16.084673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084642 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.084908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084685 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.084908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084736 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.084908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084808 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.084908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084840 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\" (UniqueName: \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.084908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.084872 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\") pod \"f2db461e-e1be-425f-a566-007bd63b5cbb\" (UID: \"f2db461e-e1be-425f-a566-007bd63b5cbb\") " Oct 31 00:43:16.085194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.085122 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:16.085369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.085344 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:16.087136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.087109 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:16.087279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.087250 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "kube-api-access-ft5th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:16.087518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.087498 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:16.087632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.087521 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret" pod "f2db461e-e1be-425f-a566-007bd63b5cbb" (UID: "f2db461e-e1be-425f-a566-007bd63b5cbb"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:16.186019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.185981 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.186019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.186011 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft5th\" (UniqueName: \"kubernetes.io/projected/f2db461e-e1be-425f-a566-007bd63b5cbb-kube-api-access-ft5th\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.186019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.186022 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.186019 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.186030 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.186341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.186039 5577 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-test-usage-logs-loki-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.186341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.186049 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/f2db461e-e1be-425f-a566-007bd63b5cbb-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:16.801395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.801357 5577 generic.go:356] "Generic (PLEG): container finished" podID="f2db461e-e1be-425f-a566-007bd63b5cbb" containerID="8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac" exitCode=0 Oct 31 00:43:16.801903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.801401 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" Oct 31 00:43:16.801903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.801416 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" event={"ID":"f2db461e-e1be-425f-a566-007bd63b5cbb","Type":"ContainerDied","Data":"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac"} Oct 31 00:43:16.801903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.801456 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79" event={"ID":"f2db461e-e1be-425f-a566-007bd63b5cbb","Type":"ContainerDied","Data":"c6d30ee161c638ee7969be4bdaa30e72eff762069f8791b981461a645231d51f"} Oct 31 00:43:16.801903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.801480 5577 scope.go:122] "RemoveContainer" containerID="8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac" Oct 31 00:43:16.811099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.811078 5577 scope.go:122] "RemoveContainer" containerID="8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac" Oct 31 00:43:16.811381 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:16.811361 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac\": container with ID starting with 8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac not found: ID does not exist" containerID="8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac" Oct 31 00:43:16.811432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.811391 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac"} err="failed to get container status \"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac\": rpc error: code = NotFound desc = could not find container \"8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac\": container with ID starting with 8179c7783cec764e6a6d44d178945ffb0a202d32d219c5c8c50a78085aa320ac not found: ID does not exist" Oct 31 00:43:16.823057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.823033 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:43:16.827176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:16.827154 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-clj79"] Oct 31 00:43:17.129394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:17.129317 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2db461e-e1be-425f-a566-007bd63b5cbb" path="/var/lib/kubelet/pods/f2db461e-e1be-425f-a566-007bd63b5cbb/volumes" Oct 31 00:43:21.688118 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:21.688083 5577 scope.go:122] "RemoveContainer" containerID="6c2d0eb193e8acdea9aaa559b9c2e031fc4d30703ea7ba577331bf2e3be486f5" Oct 31 00:43:26.867367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:26.867332 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:28.772477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:28.772447 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:29.292130 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:29.292096 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:43:55.185948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.185920 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186252 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f72f403b-78fe-467c-afed-60dcfaa04e74" containerName="collector" Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186263 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f72f403b-78fe-467c-afed-60dcfaa04e74" containerName="collector" Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186281 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f2db461e-e1be-425f-a566-007bd63b5cbb" containerName="loki-querier" Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186288 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2db461e-e1be-425f-a566-007bd63b5cbb" containerName="loki-querier" Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186336 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f72f403b-78fe-467c-afed-60dcfaa04e74" containerName="collector" Oct 31 00:43:55.186402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.186346 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2db461e-e1be-425f-a566-007bd63b5cbb" containerName="loki-querier" Oct 31 00:43:55.189181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.189160 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.201046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.201018 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:55.257673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.257642 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:43:55.257871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.257682 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rb96t\"" Oct 31 00:43:55.262305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.262273 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.278685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.278661 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:43:55.285001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.284971 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.285141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.285012 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvsj8\" (UniqueName: \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.285141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.285123 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.385696 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.385659 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.385900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.385747 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.385900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.385786 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvsj8\" (UniqueName: \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.386394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.386375 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.388222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.388202 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.390917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.390889 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:55.398501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.398482 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvsj8\" (UniqueName: \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\") pod \"data-science-usage-logs-collector-5b5b9f5c4f-tvl69\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.505769 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.505731 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:55.783950 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.783877 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:55.787245 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:43:55.787218 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3f4945_e773_4364_b91f_397c529faf10.slice/crio-e508493faff854cffca5889d981fe9e69ef31ccf0370eeec163160a91c3f7f0e WatchSource:0}: Error finding container e508493faff854cffca5889d981fe9e69ef31ccf0370eeec163160a91c3f7f0e: Status 404 returned error can't find the container with id e508493faff854cffca5889d981fe9e69ef31ccf0370eeec163160a91c3f7f0e Oct 31 00:43:55.789895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.789860 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:55.790002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.789922 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:43:55.960557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.960527 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" event={"ID":"cb3f4945-e773-4364-b91f-397c529faf10","Type":"ContainerStarted","Data":"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966"} Oct 31 00:43:55.960731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.960574 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" event={"ID":"cb3f4945-e773-4364-b91f-397c529faf10","Type":"ContainerStarted","Data":"e508493faff854cffca5889d981fe9e69ef31ccf0370eeec163160a91c3f7f0e"} Oct 31 00:43:55.981162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:55.981109 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" podStartSLOduration=0.981095986 podStartE2EDuration="981.095986ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:55.980962343 +0000 UTC m=+2674.972621946" watchObservedRunningTime="2026-10-31 00:43:55.981095986 +0000 UTC m=+2674.972755569" Oct 31 00:43:56.677255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.677216 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.831293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.831265 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:43:56.831488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.831435 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="2a194581-246e-4476-b18b-baa47d436613" containerName="loki-compactor" containerID="cri-o://b131dd2c26dbdd912f0e64d570fc4319fda04061182841c38737261d7978e4ab" gracePeriod=30 Oct 31 00:43:56.841806 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.841773 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:56.861150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.861096 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:43:56.861427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.861310 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" podUID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" containerName="loki-query-frontend" containerID="cri-o://b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919" gracePeriod=30 Oct 31 00:43:56.891818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.891789 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:43:56.892018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.891996 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" podUID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" containerName="loki-distributor" containerID="cri-o://f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a" gracePeriod=30 Oct 31 00:43:56.904518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.904491 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:56.964685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.964599 5577 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" secret="" err="secret \"data-science-usage-logs-collector-dockercfg-rb96t\" not found" Oct 31 00:43:56.986971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:56.986939 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:43:57.109428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.109405 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.109428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.109424 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.109597 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.109478 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal podName:cb3f4945-e773-4364-b91f-397c529faf10 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:57.609462387 +0000 UTC m=+2676.601121949 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-tvl69" (UID: "cb3f4945-e773-4364-b91f-397c529faf10") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.109597 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.109494 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs podName:cb3f4945-e773-4364-b91f-397c529faf10 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:57.60948605 +0000 UTC m=+2676.601145612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-tvl69" (UID: "cb3f4945-e773-4364-b91f-397c529faf10") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.110333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.110319 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:43:57.210020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.209988 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\") pod \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " Oct 31 00:43:57.210194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210029 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\") pod \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " Oct 31 00:43:57.210194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210068 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " Oct 31 00:43:57.210194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210124 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\" (UniqueName: \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\") pod \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " Oct 31 00:43:57.210194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210162 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\") pod \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\" (UID: \"b66a96d4-3035-4caa-b2b3-46b02d43d97f\") " Oct 31 00:43:57.210604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210574 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config" pod "b66a96d4-3035-4caa-b2b3-46b02d43d97f" (UID: "b66a96d4-3035-4caa-b2b3-46b02d43d97f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.210771 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.210596 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle" pod "b66a96d4-3035-4caa-b2b3-46b02d43d97f" (UID: "b66a96d4-3035-4caa-b2b3-46b02d43d97f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:57.212422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.212396 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw" pod "b66a96d4-3035-4caa-b2b3-46b02d43d97f" (UID: "b66a96d4-3035-4caa-b2b3-46b02d43d97f"). InnerVolumeSpecName "kube-api-access-v6srw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:57.212519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.212434 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http" pod "b66a96d4-3035-4caa-b2b3-46b02d43d97f" (UID: "b66a96d4-3035-4caa-b2b3-46b02d43d97f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.212519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.212478 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc" pod "b66a96d4-3035-4caa-b2b3-46b02d43d97f" (UID: "b66a96d4-3035-4caa-b2b3-46b02d43d97f"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:57.311112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.311078 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.311112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.311106 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.311112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.311116 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-query-frontend-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.311350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.311125 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6srw\" (UniqueName: \"kubernetes.io/projected/b66a96d4-3035-4caa-b2b3-46b02d43d97f-kube-api-access-v6srw\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.311350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.311136 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b66a96d4-3035-4caa-b2b3-46b02d43d97f-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:57.613893 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.613859 5577 configmap.go:193] Couldn't get configMap redhat-ods-monitoring/data-science-usage-logs-collector-9d8884b6: configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.614074 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.613915 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-usage-logs-collector-headless-tls: secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.614074 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.613927 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal podName:cb3f4945-e773-4364-b91f-397c529faf10 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:58.613913661 +0000 UTC m=+2677.605573236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otc-internal" (UniqueName: "kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal") pod "data-science-usage-logs-collector-5b5b9f5c4f-tvl69" (UID: "cb3f4945-e773-4364-b91f-397c529faf10") : configmap "data-science-usage-logs-collector-9d8884b6" not found Oct 31 00:43:57.614074 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.613987 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs podName:cb3f4945-e773-4364-b91f-397c529faf10 nodeName:}" failed. No retries permitted until 2026-10-31 00:43:58.613968183 +0000 UTC m=+2677.605627745 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs") pod "data-science-usage-logs-collector-5b5b9f5c4f-tvl69" (UID: "cb3f4945-e773-4364-b91f-397c529faf10") : secret "data-science-usage-logs-collector-headless-tls" not found Oct 31 00:43:57.970108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.970077 5577 generic.go:356] "Generic (PLEG): container finished" podID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" containerID="b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919" exitCode=0 Oct 31 00:43:57.970581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.970150 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" Oct 31 00:43:57.970581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.970160 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" event={"ID":"b66a96d4-3035-4caa-b2b3-46b02d43d97f","Type":"ContainerDied","Data":"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919"} Oct 31 00:43:57.970581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.970199 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2" event={"ID":"b66a96d4-3035-4caa-b2b3-46b02d43d97f","Type":"ContainerDied","Data":"54d386219c4ef6f1caa67b44432a2089e75c1226bee0dc5c8617e039c788b86a"} Oct 31 00:43:57.970581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.970217 5577 scope.go:122] "RemoveContainer" containerID="b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919" Oct 31 00:43:57.972334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.972311 5577 generic.go:356] "Generic (PLEG): container finished" podID="2a194581-246e-4476-b18b-baa47d436613" containerID="b131dd2c26dbdd912f0e64d570fc4319fda04061182841c38737261d7978e4ab" exitCode=0 Oct 31 00:43:57.972463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.972394 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"2a194581-246e-4476-b18b-baa47d436613","Type":"ContainerDied","Data":"b131dd2c26dbdd912f0e64d570fc4319fda04061182841c38737261d7978e4ab"} Oct 31 00:43:57.972534 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.972514 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" podUID="cb3f4945-e773-4364-b91f-397c529faf10" containerName="otc-container" containerID="cri-o://9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966" gracePeriod=30 Oct 31 00:43:57.979504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.979485 5577 scope.go:122] "RemoveContainer" containerID="b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919" Oct 31 00:43:57.979861 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:57.979835 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919\": container with ID starting with b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919 not found: ID does not exist" containerID="b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919" Oct 31 00:43:57.979940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:57.979874 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919"} err="failed to get container status \"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919\": rpc error: code = NotFound desc = could not find container \"b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919\": container with ID starting with b3f45565285199a67e85f6b82f3b34a80fee8ea1c24810fbec2c844e69a1d919 not found: ID does not exist" Oct 31 00:43:58.007462 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.007438 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:43:58.012738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.012689 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbrpbq2"] Oct 31 00:43:58.092301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.092278 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:58.208270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.208243 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:58.219967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.219940 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220096 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.219995 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220091 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220126 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220173 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220206 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220251 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\" (UniqueName: \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\") pod \"2a194581-246e-4476-b18b-baa47d436613\" (UID: \"2a194581-246e-4476-b18b-baa47d436613\") " Oct 31 00:43:58.220347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220273 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.220599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220537 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.220993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220941 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.220993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.220968 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2a194581-246e-4476-b18b-baa47d436613-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.222528 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.222499 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.222875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.222850 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.222875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.222865 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.222875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.222868 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "kube-api-access-llfrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.223052 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.222982 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca" pod "2a194581-246e-4476-b18b-baa47d436613" (UID: "2a194581-246e-4476-b18b-baa47d436613"). InnerVolumeSpecName "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:43:58.321659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.321621 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\") pod \"cb3f4945-e773-4364-b91f-397c529faf10\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " Oct 31 00:43:58.321659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.321657 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\") pod \"cb3f4945-e773-4364-b91f-397c529faf10\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " Oct 31 00:43:58.321887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.321688 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\" (UniqueName: \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\") pod \"cb3f4945-e773-4364-b91f-397c529faf10\" (UID: \"cb3f4945-e773-4364-b91f-397c529faf10\") " Oct 31 00:43:58.322024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322004 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llfrk\" (UniqueName: \"kubernetes.io/projected/2a194581-246e-4476-b18b-baa47d436613-kube-api-access-llfrk\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.322067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322032 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.322067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322032 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal" pod "cb3f4945-e773-4364-b91f-397c529faf10" (UID: "cb3f4945-e773-4364-b91f-397c529faf10"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:43:58.322132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322070 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:43:58.322132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322085 5577 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-test-loki-lifecycle-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.322132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.322098 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/2a194581-246e-4476-b18b-baa47d436613-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.324024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.324001 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8" pod "cb3f4945-e773-4364-b91f-397c529faf10" (UID: "cb3f4945-e773-4364-b91f-397c529faf10"). InnerVolumeSpecName "kube-api-access-jvsj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:43:58.324094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.324052 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs" pod "cb3f4945-e773-4364-b91f-397c529faf10" (UID: "cb3f4945-e773-4364-b91f-397c529faf10"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:43:58.330589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.330572 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:43:58.423039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.423012 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/cb3f4945-e773-4364-b91f-397c529faf10-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.423039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.423039 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvsj8\" (UniqueName: \"kubernetes.io/projected/cb3f4945-e773-4364-b91f-397c529faf10-kube-api-access-jvsj8\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.423187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.423054 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-142ae4646c2c18dcb6978f1388429458a476710589bcb026ecd45ad140ae0094\"" Oct 31 00:43:58.423187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.423066 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/cb3f4945-e773-4364-b91f-397c529faf10-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:43:58.979112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.979078 5577 generic.go:356] "Generic (PLEG): container finished" podID="cb3f4945-e773-4364-b91f-397c529faf10" containerID="9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966" exitCode=0 Oct 31 00:43:58.979583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.979126 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" Oct 31 00:43:58.979583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.979163 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" event={"ID":"cb3f4945-e773-4364-b91f-397c529faf10","Type":"ContainerDied","Data":"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966"} Oct 31 00:43:58.979583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.979215 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69" event={"ID":"cb3f4945-e773-4364-b91f-397c529faf10","Type":"ContainerDied","Data":"e508493faff854cffca5889d981fe9e69ef31ccf0370eeec163160a91c3f7f0e"} Oct 31 00:43:58.979583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.979243 5577 scope.go:122] "RemoveContainer" containerID="9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966" Oct 31 00:43:58.980607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.980586 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"2a194581-246e-4476-b18b-baa47d436613","Type":"ContainerDied","Data":"4174fdc1c85a251a6aa91de912c1078d6bfd4a0e48b2a53a5733643b55ab7c56"} Oct 31 00:43:58.980607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.980602 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:43:58.988796 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.988774 5577 scope.go:122] "RemoveContainer" containerID="9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966" Oct 31 00:43:58.989083 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:58.989056 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966\": container with ID starting with 9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966 not found: ID does not exist" containerID="9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966" Oct 31 00:43:58.989175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.989083 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966"} err="failed to get container status \"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966\": rpc error: code = NotFound desc = could not find container \"9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966\": container with ID starting with 9ac3c318db29aae7d2e4b8ead771b27653ac17e879c50d1ef922f166fe18f966 not found: ID does not exist" Oct 31 00:43:58.989175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:58.989098 5577 scope.go:122] "RemoveContainer" containerID="b131dd2c26dbdd912f0e64d570fc4319fda04061182841c38737261d7978e4ab" Oct 31 00:43:59.005181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.005159 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:59.009296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.009277 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5b5b9f5c4f-tvl69"] Oct 31 00:43:59.021955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.021931 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:43:59.026857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.026840 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:43:59.129380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.129345 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a194581-246e-4476-b18b-baa47d436613" path="/var/lib/kubelet/pods/2a194581-246e-4476-b18b-baa47d436613/volumes" Oct 31 00:43:59.129811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.129797 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" path="/var/lib/kubelet/pods/b66a96d4-3035-4caa-b2b3-46b02d43d97f/volumes" Oct 31 00:43:59.130131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:43:59.130119 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3f4945-e773-4364-b91f-397c529faf10" path="/var/lib/kubelet/pods/cb3f4945-e773-4364-b91f-397c529faf10/volumes" Oct 31 00:43:59.291432 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:43:59.291397 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:44:00.234149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.234127 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:44:00.339277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339209 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\") pod \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " Oct 31 00:44:00.339277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339254 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\") pod \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " Oct 31 00:44:00.339442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339287 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\") pod \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " Oct 31 00:44:00.339442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339322 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\" (UniqueName: \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\") pod \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " Oct 31 00:44:00.339442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339356 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\") pod \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\" (UID: \"ad7fb426-0d27-43ce-b11e-e8a917c737e0\") " Oct 31 00:44:00.339643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339616 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config" pod "ad7fb426-0d27-43ce-b11e-e8a917c737e0" (UID: "ad7fb426-0d27-43ce-b11e-e8a917c737e0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:00.339768 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.339746 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle" pod "ad7fb426-0d27-43ce-b11e-e8a917c737e0" (UID: "ad7fb426-0d27-43ce-b11e-e8a917c737e0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:00.341583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.341553 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc" pod "ad7fb426-0d27-43ce-b11e-e8a917c737e0" (UID: "ad7fb426-0d27-43ce-b11e-e8a917c737e0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:00.341685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.341619 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b" pod "ad7fb426-0d27-43ce-b11e-e8a917c737e0" (UID: "ad7fb426-0d27-43ce-b11e-e8a917c737e0"). InnerVolumeSpecName "kube-api-access-2hv9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:00.341685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.341627 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http" pod "ad7fb426-0d27-43ce-b11e-e8a917c737e0" (UID: "ad7fb426-0d27-43ce-b11e-e8a917c737e0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:00.440174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.440141 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:00.440174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.440170 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:00.440174 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.440182 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:00.440394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.440192 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hv9b\" (UniqueName: \"kubernetes.io/projected/ad7fb426-0d27-43ce-b11e-e8a917c737e0-kube-api-access-2hv9b\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:00.440394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.440202 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/ad7fb426-0d27-43ce-b11e-e8a917c737e0-data-science-usage-logs-lokistack-distributor-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:00.682629 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682555 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:44:00.682871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682857 5577 cpu_manager.go:414] "containerMap: removing container" podUID="cb3f4945-e773-4364-b91f-397c529faf10" containerName="otc-container" Oct 31 00:44:00.682911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682873 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3f4945-e773-4364-b91f-397c529faf10" containerName="otc-container" Oct 31 00:44:00.682911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682887 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" containerName="loki-distributor" Oct 31 00:44:00.682911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682892 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" containerName="loki-distributor" Oct 31 00:44:00.682911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682903 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" containerName="loki-query-frontend" Oct 31 00:44:00.682911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682908 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" containerName="loki-query-frontend" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682914 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2a194581-246e-4476-b18b-baa47d436613" containerName="loki-compactor" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682919 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a194581-246e-4476-b18b-baa47d436613" containerName="loki-compactor" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682989 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3f4945-e773-4364-b91f-397c529faf10" containerName="otc-container" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.682998 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" containerName="loki-distributor" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.683005 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66a96d4-3035-4caa-b2b3-46b02d43d97f" containerName="loki-query-frontend" Oct 31 00:44:00.683090 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.683013 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a194581-246e-4476-b18b-baa47d436613" containerName="loki-compactor" Oct 31 00:44:00.686046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.686030 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.690596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.690566 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-lt9m8\"" Oct 31 00:44:00.697535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.697484 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:44:00.724184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.724145 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:44:00.727256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.727233 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.740184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.740154 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:44:00.759676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.759639 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.791061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.791035 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 31 00:44:00.791222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.791112 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-grpc\"" Oct 31 00:44:00.791222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.791136 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ingester-http\"" Oct 31 00:44:00.818184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.818149 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:44:00.821404 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.821382 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.831925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.831896 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:44:00.843010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.842981 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.843147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843028 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.843147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843064 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.843242 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843162 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.843288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843240 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.843321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843281 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.843372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843330 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.843516 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843479 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.843601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843578 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsjnj\" (UniqueName: \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.843669 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843618 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.843764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.843677 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zh7\" (UniqueName: \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.857920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.857890 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:44:00.865915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.865891 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.872666 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.872619 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:44:00.891939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.891910 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 31 00:44:00.891939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.891911 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-grpc\"" Oct 31 00:44:00.892183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.892015 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 31 00:44:00.892183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.892083 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-http\"" Oct 31 00:44:00.901315 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.901289 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:00.944309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944223 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.944309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944269 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.944494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944344 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.944494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944382 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.944494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944402 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.944494 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944476 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsjnj\" (UniqueName: \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.944672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944524 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.944672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944550 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zh7\" (UniqueName: \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.944672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944571 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.944672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944605 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.944900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944665 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfkwp\" (UniqueName: \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.944900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944812 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.944900 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944863 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.945054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944899 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.945054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.944930 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.945054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945007 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.945207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945045 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.945207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945098 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gchsf\" (UniqueName: \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.945207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945171 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.945372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945207 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.945372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945239 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:00.945372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945271 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:00.945372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945315 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.945592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945565 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.945906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.945882 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.946181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.946082 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.947868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.947848 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.947966 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.947947 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.948028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.947965 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.948064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.948053 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.948102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.948062 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.958843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.958817 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zh7\" (UniqueName: \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:00.959017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.959000 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsjnj\" (UniqueName: \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\") pod \"data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:00.991743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.991694 5577 generic.go:356] "Generic (PLEG): container finished" podID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" containerID="f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a" exitCode=0 Oct 31 00:44:00.991905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.991740 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" event={"ID":"ad7fb426-0d27-43ce-b11e-e8a917c737e0","Type":"ContainerDied","Data":"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a"} Oct 31 00:44:00.991905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.991771 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" Oct 31 00:44:00.991905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.991792 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll" event={"ID":"ad7fb426-0d27-43ce-b11e-e8a917c737e0","Type":"ContainerDied","Data":"dd5493f5656a6c62c8c87e2379e1e02ef5e27c44da9ea4e070f1e38602fbe196"} Oct 31 00:44:00.991905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.991814 5577 scope.go:122] "RemoveContainer" containerID="f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a" Oct 31 00:44:00.997412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:00.997390 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:01.004510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.004439 5577 scope.go:122] "RemoveContainer" containerID="f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a" Oct 31 00:44:01.005372 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:44:01.005017 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a\": container with ID starting with f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a not found: ID does not exist" containerID="f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a" Oct 31 00:44:01.005372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.005056 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a"} err="failed to get container status \"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a\": rpc error: code = NotFound desc = could not find container \"f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a\": container with ID starting with f0a82e9b995e087a5e6d64a0fe37367f95e27bf32c9d5dd5ac7c4ba394ec2a4a not found: ID does not exist" Oct 31 00:44:01.025193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.025169 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:44:01.027231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.027209 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-j27ll"] Oct 31 00:44:01.044848 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.044819 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.046803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046782 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.046871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046828 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.046871 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046854 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfkwp\" (UniqueName: \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.046944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046899 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.046944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046934 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gchsf\" (UniqueName: \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.047011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.046982 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.047051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.047014 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.047085 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.047065 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.047116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.047101 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.047176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.047138 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.047216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.047196 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.048238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.048216 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.048470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.048448 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.048973 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.048935 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.049093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.048990 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.051524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.051498 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.057830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.057802 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.059203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.059170 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.059835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.059810 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.060248 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.060220 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.062621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.062595 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfkwp\" (UniqueName: \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:01.065585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.065567 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gchsf\" (UniqueName: \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.129081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.129046 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad7fb426-0d27-43ce-b11e-e8a917c737e0" path="/var/lib/kubelet/pods/ad7fb426-0d27-43ce-b11e-e8a917c737e0/volumes" Oct 31 00:44:01.173062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.173033 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.179450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.179425 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:01.277620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.277583 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:44:01.280422 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:44:01.280394 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0ccc22fa_fa78_4af3_8413_02e72fdf1bde.slice/crio-bdb2eb639e3eac578db907df39c83e3f9d38110abd7cb8874157be3771c44a86 WatchSource:0}: Error finding container bdb2eb639e3eac578db907df39c83e3f9d38110abd7cb8874157be3771c44a86: Status 404 returned error can't find the container with id bdb2eb639e3eac578db907df39c83e3f9d38110abd7cb8874157be3771c44a86 Oct 31 00:44:01.303994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.303957 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:01.455471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.455439 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:44:01.459183 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:44:01.459155 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64beb84a_7b51_4890_b49e_3d154836ffc0.slice/crio-9eae38e94547be350a29cf00bf72b8a0c2520c1a77020de9dd94ac8a5450034c WatchSource:0}: Error finding container 9eae38e94547be350a29cf00bf72b8a0c2520c1a77020de9dd94ac8a5450034c: Status 404 returned error can't find the container with id 9eae38e94547be350a29cf00bf72b8a0c2520c1a77020de9dd94ac8a5450034c Oct 31 00:44:01.997936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.997899 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" event={"ID":"0ccc22fa-fa78-4af3-8413-02e72fdf1bde","Type":"ContainerStarted","Data":"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a"} Oct 31 00:44:01.998151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.997951 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:01.998151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:01.997969 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" event={"ID":"0ccc22fa-fa78-4af3-8413-02e72fdf1bde","Type":"ContainerStarted","Data":"bdb2eb639e3eac578db907df39c83e3f9d38110abd7cb8874157be3771c44a86"} Oct 31 00:44:02.000264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:02.000240 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" event={"ID":"64beb84a-7b51-4890-b49e-3d154836ffc0","Type":"ContainerStarted","Data":"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1"} Oct 31 00:44:02.000383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:02.000269 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" event={"ID":"64beb84a-7b51-4890-b49e-3d154836ffc0","Type":"ContainerStarted","Data":"9eae38e94547be350a29cf00bf72b8a0c2520c1a77020de9dd94ac8a5450034c"} Oct 31 00:44:02.000383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:02.000348 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:02.017577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:02.017523 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" podStartSLOduration=2.01751153 podStartE2EDuration="2.01751153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:02.017451254 +0000 UTC m=+2681.009110839" watchObservedRunningTime="2026-10-31 00:44:02.01751153 +0000 UTC m=+2681.009171114" Oct 31 00:44:02.038472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:02.038425 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" podStartSLOduration=2.038411895 podStartE2EDuration="2.038411895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:02.037627202 +0000 UTC m=+2681.029286786" watchObservedRunningTime="2026-10-31 00:44:02.038411895 +0000 UTC m=+2681.030071469" Oct 31 00:44:06.595890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.595861 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.603365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.603332 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.696439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.696402 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.696588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.696536 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.700550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.700530 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") DevicePath \"csi-142ae4646c2c18dcb6978f1388429458a476710589bcb026ecd45ad140ae0094\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.794724 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.794680 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/ce8a44ea6fd624fabb855523df239646d3dcc3258d2c5f84025b15690a2664c7/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.809031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.809000 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:06.833005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:06.832983 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:07.106119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:07.106093 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:44:07.108604 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:44:07.108559 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29bf08b_ce42_4961_a9bd_f45f6866d341.slice/crio-bb1b0c671001392224048e78e2528d63b66fc7e66751fe18ee725eecac32434a WatchSource:0}: Error finding container bb1b0c671001392224048e78e2528d63b66fc7e66751fe18ee725eecac32434a: Status 404 returned error can't find the container with id bb1b0c671001392224048e78e2528d63b66fc7e66751fe18ee725eecac32434a Oct 31 00:44:08.026170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:08.026133 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"c29bf08b-ce42-4961-a9bd-f45f6866d341","Type":"ContainerStarted","Data":"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615"} Oct 31 00:44:08.026170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:08.026174 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"c29bf08b-ce42-4961-a9bd-f45f6866d341","Type":"ContainerStarted","Data":"bb1b0c671001392224048e78e2528d63b66fc7e66751fe18ee725eecac32434a"} Oct 31 00:44:08.026606 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:08.026253 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:08.048491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:08.048445 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=8.048430752 podStartE2EDuration="8.048430752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:08.047699016 +0000 UTC m=+2687.039358600" watchObservedRunningTime="2026-10-31 00:44:08.048430752 +0000 UTC m=+2687.040090336" Oct 31 00:44:10.227747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.227692 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.234884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.234846 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.328691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.328640 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.328903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.328801 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.332556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.332537 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") DevicePath \"csi-c6e18bffb84f40731f2893c8d5f46a40e75fd71fbc8c85a33e146c9fb6d9e49c\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.425222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.425191 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/7c9d44c2ac5b185f6db2eab8b6cc11f2151bd1a519bd148465043863d898a7a3/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:10.445757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:10.445729 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.455460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.455421 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.462214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.462192 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.556902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.556864 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.557056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.556975 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.560684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.560663 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") DevicePath \"csi-67b10419277b12273e208305fa2a99f21576a4d00bbe4ed915d7f356dddfe7fa\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.653131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.653103 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/fdf3d44672bff7110481cf42cbe6e99aa39e5c8305a671c1179f840d2ab17bcc/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.666692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.666665 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"data-science-usage-logs-lokistack-ingester-0\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:13.939154 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:13.939121 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:14.215361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:14.215289 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:44:14.218725 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:44:14.218680 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5fd0611_f013_491d_a0db_d3df5f2c38de.slice/crio-83d2c38a1b768ea01787c0cbcc693c5dcb9fd178dcde28f89a17d46017a407be WatchSource:0}: Error finding container 83d2c38a1b768ea01787c0cbcc693c5dcb9fd178dcde28f89a17d46017a407be: Status 404 returned error can't find the container with id 83d2c38a1b768ea01787c0cbcc693c5dcb9fd178dcde28f89a17d46017a407be Oct 31 00:44:15.059760 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:15.059725 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"d5fd0611-f013-491d-a0db-d3df5f2c38de","Type":"ContainerStarted","Data":"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a"} Oct 31 00:44:15.060193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:15.059768 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"d5fd0611-f013-491d-a0db-d3df5f2c38de","Type":"ContainerStarted","Data":"83d2c38a1b768ea01787c0cbcc693c5dcb9fd178dcde28f89a17d46017a407be"} Oct 31 00:44:15.060193 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:15.059827 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:15.082677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:15.082628 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podStartSLOduration=15.082612656 podStartE2EDuration="15.082612656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:15.081771655 +0000 UTC m=+2694.073431260" watchObservedRunningTime="2026-10-31 00:44:15.082612656 +0000 UTC m=+2694.074272240" Oct 31 00:44:21.398685 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:21.398653 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:44:21.405111 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:21.405090 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:44:23.009851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:23.009821 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:44:23.010265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:23.010204 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:44:26.918834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:26.918797 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.038614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:29.038535 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:44:29.291764 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:44:29.291667 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:44:29.708963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:29.708878 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:29.837149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:29.837112 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:44:36.069868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:36.069834 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:46.068363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:46.068323 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:44:56.068883 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:44:56.068850 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:44:59.292024 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:44:59.291986 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:45:00.387577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.387543 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:00.391309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.391284 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.402777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.402681 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:00.448050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.448014 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.448233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.448138 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.448301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.448238 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.452686 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.452660 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.463701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.463677 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 31 00:45:00.463830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.463778 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-gd7lc\"" Oct 31 00:45:00.482599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.482576 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 31 00:45:00.549148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.549118 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.549274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.549167 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.549274 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.549201 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.549981 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.549914 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.551854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.551831 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.562244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.562225 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\") pod \"data-science-usage-logs-collector-76bd85dc9-zz28d\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.585972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.585932 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:00.705390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.705307 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:00.985991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.985770 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:00.988606 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:00.988576 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8376747b_83f3_42ac_8065_d225284e6727.slice/crio-efbd6577387d5d0684c7ba24154e6980639eb056a524bbd81f10d8b1fb8849e6 WatchSource:0}: Error finding container efbd6577387d5d0684c7ba24154e6980639eb056a524bbd81f10d8b1fb8849e6: Status 404 returned error can't find the container with id efbd6577387d5d0684c7ba24154e6980639eb056a524bbd81f10d8b1fb8849e6 Oct 31 00:45:00.991015 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.990983 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:00.991093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:00.991043 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:01.248576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:01.248485 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" event={"ID":"8376747b-83f3-42ac-8065-d225284e6727","Type":"ContainerStarted","Data":"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935"} Oct 31 00:45:01.248576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:01.248542 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" event={"ID":"8376747b-83f3-42ac-8065-d225284e6727","Type":"ContainerStarted","Data":"efbd6577387d5d0684c7ba24154e6980639eb056a524bbd81f10d8b1fb8849e6"} Oct 31 00:45:01.267677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:01.267592 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" podStartSLOduration=1.267573346 podStartE2EDuration="1.267573346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:01.267506262 +0000 UTC m=+2740.259165847" watchObservedRunningTime="2026-10-31 00:45:01.267573346 +0000 UTC m=+2740.259232929" Oct 31 00:45:02.472101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.471753 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.518567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.518531 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:45:02.519070 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.519038 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="c29bf08b-ce42-4961-a9bd-f45f6866d341" containerName="loki-compactor" containerID="cri-o://bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615" gracePeriod=30 Oct 31 00:45:02.528951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.528923 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:45:02.529103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.529083 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" containerID="cri-o://e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a" gracePeriod=600 Oct 31 00:45:02.592700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.592668 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:45:02.592927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.592902 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" containerID="cri-o://fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a" gracePeriod=30 Oct 31 00:45:02.600690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.600659 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:45:02.600963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.600887 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" podUID="64beb84a-7b51-4890-b49e-3d154836ffc0" containerName="loki-query-frontend" containerID="cri-o://24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1" gracePeriod=30 Oct 31 00:45:02.612147 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.612029 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:02.617473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.617445 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.752589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.752534 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:02.854972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.854944 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:45:02.970262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970228 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\") pod \"64beb84a-7b51-4890-b49e-3d154836ffc0\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " Oct 31 00:45:02.970436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970271 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"64beb84a-7b51-4890-b49e-3d154836ffc0\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " Oct 31 00:45:02.970436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970302 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\" (UniqueName: \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\") pod \"64beb84a-7b51-4890-b49e-3d154836ffc0\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " Oct 31 00:45:02.970436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970369 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\") pod \"64beb84a-7b51-4890-b49e-3d154836ffc0\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " Oct 31 00:45:02.970436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970411 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\") pod \"64beb84a-7b51-4890-b49e-3d154836ffc0\" (UID: \"64beb84a-7b51-4890-b49e-3d154836ffc0\") " Oct 31 00:45:02.970664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970627 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config" pod "64beb84a-7b51-4890-b49e-3d154836ffc0" (UID: "64beb84a-7b51-4890-b49e-3d154836ffc0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:02.970816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970773 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:02.970969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.970900 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle" pod "64beb84a-7b51-4890-b49e-3d154836ffc0" (UID: "64beb84a-7b51-4890-b49e-3d154836ffc0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:02.972596 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.972567 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc" pod "64beb84a-7b51-4890-b49e-3d154836ffc0" (UID: "64beb84a-7b51-4890-b49e-3d154836ffc0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:02.972694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.972662 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http" pod "64beb84a-7b51-4890-b49e-3d154836ffc0" (UID: "64beb84a-7b51-4890-b49e-3d154836ffc0"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:02.972694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:02.972671 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf" pod "64beb84a-7b51-4890-b49e-3d154836ffc0" (UID: "64beb84a-7b51-4890-b49e-3d154836ffc0"). InnerVolumeSpecName "kube-api-access-gchsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.008411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.008349 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:03.071297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.071267 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.071297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.071295 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gchsf\" (UniqueName: \"kubernetes.io/projected/64beb84a-7b51-4890-b49e-3d154836ffc0-kube-api-access-gchsf\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.071484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.071306 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-query-frontend-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.071484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.071316 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/64beb84a-7b51-4890-b49e-3d154836ffc0-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.258055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258015 5577 generic.go:356] "Generic (PLEG): container finished" podID="64beb84a-7b51-4890-b49e-3d154836ffc0" containerID="24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1" exitCode=0 Oct 31 00:45:03.258214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258064 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" Oct 31 00:45:03.258214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258102 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" event={"ID":"64beb84a-7b51-4890-b49e-3d154836ffc0","Type":"ContainerDied","Data":"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1"} Oct 31 00:45:03.258214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258149 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf" event={"ID":"64beb84a-7b51-4890-b49e-3d154836ffc0","Type":"ContainerDied","Data":"9eae38e94547be350a29cf00bf72b8a0c2520c1a77020de9dd94ac8a5450034c"} Oct 31 00:45:03.258214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258170 5577 scope.go:122] "RemoveContainer" containerID="24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1" Oct 31 00:45:03.258399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.258364 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" podUID="8376747b-83f3-42ac-8065-d225284e6727" containerName="otc-container" containerID="cri-o://d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935" gracePeriod=30 Oct 31 00:45:03.267332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.267308 5577 scope.go:122] "RemoveContainer" containerID="24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1" Oct 31 00:45:03.267654 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:03.267635 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1\": container with ID starting with 24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1 not found: ID does not exist" containerID="24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1" Oct 31 00:45:03.267727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.267663 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1"} err="failed to get container status \"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1\": rpc error: code = NotFound desc = could not find container \"24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1\": container with ID starting with 24e1859147a342e3d40ef91e5ab7199dcf16f69029cdc4269758f91054f865f1 not found: ID does not exist" Oct 31 00:45:03.296947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.296922 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:45:03.303061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.303039 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbkmklf"] Oct 31 00:45:03.496576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.496538 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:45:03.525776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.525752 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:03.576302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576277 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576314 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576349 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576377 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\" (UniqueName: \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576416 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576504 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576641 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\") pod \"c29bf08b-ce42-4961-a9bd-f45f6866d341\" (UID: \"c29bf08b-ce42-4961-a9bd-f45f6866d341\") " Oct 31 00:45:03.576731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576719 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.577017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.576950 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.577308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.577128 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.578690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.578667 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.578943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.578923 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "kube-api-access-bfkwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.579237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.579217 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.579289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.579238 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.579571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.579555 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca" pod "c29bf08b-ce42-4961-a9bd-f45f6866d341" (UID: "c29bf08b-ce42-4961-a9bd-f45f6866d341"). InnerVolumeSpecName "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:45:03.677916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.677885 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\") pod \"8376747b-83f3-42ac-8065-d225284e6727\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " Oct 31 00:45:03.678058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.677940 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\") pod \"8376747b-83f3-42ac-8065-d225284e6727\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " Oct 31 00:45:03.678058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678006 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\") pod \"8376747b-83f3-42ac-8065-d225284e6727\" (UID: \"8376747b-83f3-42ac-8065-d225284e6727\") " Oct 31 00:45:03.678191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678134 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfkwp\" (UniqueName: \"kubernetes.io/projected/c29bf08b-ce42-4961-a9bd-f45f6866d341-kube-api-access-bfkwp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.678191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678145 5577 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-test-loki-lifecycle-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.678191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678168 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:45:03.678191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678178 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.678191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678187 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.678397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678198 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/c29bf08b-ce42-4961-a9bd-f45f6866d341-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.678397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.678364 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal" pod "8376747b-83f3-42ac-8065-d225284e6727" (UID: "8376747b-83f3-42ac-8065-d225284e6727"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:03.680169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.680142 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs" pod "8376747b-83f3-42ac-8065-d225284e6727" (UID: "8376747b-83f3-42ac-8065-d225284e6727"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:03.680273 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.680213 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k" pod "8376747b-83f3-42ac-8065-d225284e6727" (UID: "8376747b-83f3-42ac-8065-d225284e6727"). InnerVolumeSpecName "kube-api-access-wd49k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:03.686749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.686733 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-81c40fd7-f377-4fca-a7c3-6d14328be212" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:45:03.779370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.779309 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-81c40fd7-f377-4fca-a7c3-6d14328be212\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-04661f00a1b2c11ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-142ae4646c2c18dcb6978f1388429458a476710589bcb026ecd45ad140ae0094\"" Oct 31 00:45:03.779370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.779334 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8376747b-83f3-42ac-8065-d225284e6727-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.779370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.779345 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd49k\" (UniqueName: \"kubernetes.io/projected/8376747b-83f3-42ac-8065-d225284e6727-kube-api-access-wd49k\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.779370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.779353 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8376747b-83f3-42ac-8065-d225284e6727-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:03.978112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:03.978088 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:45:04.081945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.081864 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.081945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.081908 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082004 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082028 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\" (UniqueName: \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082052 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082085 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082152 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082221 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\") pod \"d5fd0611-f013-491d-a0db-d3df5f2c38de\" (UID: \"d5fd0611-f013-491d-a0db-d3df5f2c38de\") " Oct 31 00:45:04.082432 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082398 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:04.082553 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082480 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:04.082553 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.082488 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.084598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.084575 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "kube-api-access-b4zh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:04.084797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.084739 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:04.085069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.085045 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:04.085278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.085251 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ingester-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:04.085524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.085509 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "pvc-e618b476-df27-42e1-92f9-d875e765a448". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:45:04.085576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.085522 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c" pod "d5fd0611-f013-491d-a0db-d3df5f2c38de" (UID: "d5fd0611-f013-491d-a0db-d3df5f2c38de"). InnerVolumeSpecName "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:45:04.183667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183645 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:45:04.183667 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183666 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4zh7\" (UniqueName: \"kubernetes.io/projected/d5fd0611-f013-491d-a0db-d3df5f2c38de-kube-api-access-b4zh7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.183914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183677 5577 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-test-loki-lifecycle-secret\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.183914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183686 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d5fd0611-f013-491d-a0db-d3df5f2c38de-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.183914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183700 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:45:04.183914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183729 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-grpc\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.183914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.183741 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ingester-http\" (UniqueName: \"kubernetes.io/secret/d5fd0611-f013-491d-a0db-d3df5f2c38de-data-science-usage-logs-lokistack-ingester-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:04.191853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.191839 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:45:04.196510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.196494 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-e618b476-df27-42e1-92f9-d875e765a448" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:45:04.264588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.264554 5577 generic.go:356] "Generic (PLEG): container finished" podID="8376747b-83f3-42ac-8065-d225284e6727" containerID="d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935" exitCode=0 Oct 31 00:45:04.264755 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.264600 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" Oct 31 00:45:04.264755 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.264598 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" event={"ID":"8376747b-83f3-42ac-8065-d225284e6727","Type":"ContainerDied","Data":"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935"} Oct 31 00:45:04.264896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.264759 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d" event={"ID":"8376747b-83f3-42ac-8065-d225284e6727","Type":"ContainerDied","Data":"efbd6577387d5d0684c7ba24154e6980639eb056a524bbd81f10d8b1fb8849e6"} Oct 31 00:45:04.264896 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.264789 5577 scope.go:122] "RemoveContainer" containerID="d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935" Oct 31 00:45:04.265998 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.265971 5577 generic.go:356] "Generic (PLEG): container finished" podID="c29bf08b-ce42-4961-a9bd-f45f6866d341" containerID="bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615" exitCode=0 Oct 31 00:45:04.266131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.266003 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"c29bf08b-ce42-4961-a9bd-f45f6866d341","Type":"ContainerDied","Data":"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615"} Oct 31 00:45:04.266131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.266042 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"c29bf08b-ce42-4961-a9bd-f45f6866d341","Type":"ContainerDied","Data":"bb1b0c671001392224048e78e2528d63b66fc7e66751fe18ee725eecac32434a"} Oct 31 00:45:04.266131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.266016 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 31 00:45:04.267451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.267429 5577 generic.go:356] "Generic (PLEG): container finished" podID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerID="e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a" exitCode=0 Oct 31 00:45:04.267551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.267461 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" Oct 31 00:45:04.267551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.267489 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"d5fd0611-f013-491d-a0db-d3df5f2c38de","Type":"ContainerDied","Data":"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a"} Oct 31 00:45:04.267551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.267516 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0" event={"ID":"d5fd0611-f013-491d-a0db-d3df5f2c38de","Type":"ContainerDied","Data":"83d2c38a1b768ea01787c0cbcc693c5dcb9fd178dcde28f89a17d46017a407be"} Oct 31 00:45:04.275671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.275652 5577 scope.go:122] "RemoveContainer" containerID="d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935" Oct 31 00:45:04.275997 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:04.275977 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935\": container with ID starting with d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935 not found: ID does not exist" containerID="d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935" Oct 31 00:45:04.276069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.276002 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935"} err="failed to get container status \"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935\": rpc error: code = NotFound desc = could not find container \"d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935\": container with ID starting with d4d215f5d99fb8aadbefb6f455047e5bd6ace56876048ea04b9ec43fc7a18935 not found: ID does not exist" Oct 31 00:45:04.276069 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.276018 5577 scope.go:122] "RemoveContainer" containerID="bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615" Oct 31 00:45:04.284469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.284453 5577 scope.go:122] "RemoveContainer" containerID="bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615" Oct 31 00:45:04.284536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.284498 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-21e03e15-9a97-42f1-bf70-38ff7cf02ac7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-08fef128f607a695c\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-67b10419277b12273e208305fa2a99f21576a4d00bbe4ed915d7f356dddfe7fa\"" Oct 31 00:45:04.284536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.284517 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-e618b476-df27-42e1-92f9-d875e765a448\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-039a993fc6abfb5d6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-c6e18bffb84f40731f2893c8d5f46a40e75fd71fbc8c85a33e146c9fb6d9e49c\"" Oct 31 00:45:04.284721 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:04.284691 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615\": container with ID starting with bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615 not found: ID does not exist" containerID="bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615" Oct 31 00:45:04.284822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.284802 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615"} err="failed to get container status \"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615\": rpc error: code = NotFound desc = could not find container \"bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615\": container with ID starting with bbbefc311b3a74cde835fe5d52a754be2af8377b5b54639d317063d06f456615 not found: ID does not exist" Oct 31 00:45:04.284866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.284826 5577 scope.go:122] "RemoveContainer" containerID="e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a" Oct 31 00:45:04.289557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.289538 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:04.292925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.292658 5577 scope.go:122] "RemoveContainer" containerID="e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a" Oct 31 00:45:04.293240 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:04.293219 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a\": container with ID starting with e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a not found: ID does not exist" containerID="e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a" Oct 31 00:45:04.293314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.293243 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a"} err="failed to get container status \"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a\": rpc error: code = NotFound desc = could not find container \"e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a\": container with ID starting with e5e8dd676b891070d4cf998df24a8d872402562a0d4388464b1d6aa16c6f4f1a not found: ID does not exist" Oct 31 00:45:04.293797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.293780 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd85dc9-zz28d"] Oct 31 00:45:04.305365 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.305347 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:45:04.311519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.311498 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 31 00:45:04.322949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.322930 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:45:04.329466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.329447 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-ingester-0"] Oct 31 00:45:04.733014 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.732979 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:45:04.737165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:04.737140 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/seaweedfs-bucket-creator"] Oct 31 00:45:05.129613 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:05.129580 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64beb84a-7b51-4890-b49e-3d154836ffc0" path="/var/lib/kubelet/pods/64beb84a-7b51-4890-b49e-3d154836ffc0/volumes" Oct 31 00:45:05.129960 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:05.129947 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8376747b-83f3-42ac-8065-d225284e6727" path="/var/lib/kubelet/pods/8376747b-83f3-42ac-8065-d225284e6727/volumes" Oct 31 00:45:05.130216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:05.130205 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1046857-f419-4b53-b1c2-7b2c20237b77" path="/var/lib/kubelet/pods/b1046857-f419-4b53-b1c2-7b2c20237b77/volumes" Oct 31 00:45:05.130869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:05.130855 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c29bf08b-ce42-4961-a9bd-f45f6866d341" path="/var/lib/kubelet/pods/c29bf08b-ce42-4961-a9bd-f45f6866d341/volumes" Oct 31 00:45:05.131216 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:05.131204 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" path="/var/lib/kubelet/pods/d5fd0611-f013-491d-a0db-d3df5f2c38de/volumes" Oct 31 00:45:13.008933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:13.008894 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:19.458112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:19.458073 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:23.009291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.009252 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:45:23.009735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.009357 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:45:23.329160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.329136 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:45:23.352658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.352623 5577 generic.go:356] "Generic (PLEG): container finished" podID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerID="fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a" exitCode=0 Oct 31 00:45:23.353648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.352674 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" Oct 31 00:45:23.353648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.352675 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" event={"ID":"0ccc22fa-fa78-4af3-8413-02e72fdf1bde","Type":"ContainerDied","Data":"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a"} Oct 31 00:45:23.353648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.352739 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt" event={"ID":"0ccc22fa-fa78-4af3-8413-02e72fdf1bde","Type":"ContainerDied","Data":"bdb2eb639e3eac578db907df39c83e3f9d38110abd7cb8874157be3771c44a86"} Oct 31 00:45:23.353648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.352763 5577 scope.go:122] "RemoveContainer" containerID="fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a" Oct 31 00:45:23.364107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.363695 5577 scope.go:122] "RemoveContainer" containerID="fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a" Oct 31 00:45:23.364219 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:23.364193 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a\": container with ID starting with fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a not found: ID does not exist" containerID="fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a" Oct 31 00:45:23.364272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.364230 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a"} err="failed to get container status \"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a\": rpc error: code = NotFound desc = could not find container \"fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a\": container with ID starting with fc3a686c245f950b54eea8ff9025b742c3dcff4a91d8b3e433c6fa3b659f365a not found: ID does not exist" Oct 31 00:45:23.441684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.441649 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\") pod \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " Oct 31 00:45:23.441862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.441697 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\") pod \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " Oct 31 00:45:23.441862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.441773 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\" (UniqueName: \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\") pod \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " Oct 31 00:45:23.441862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.441795 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\") pod \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " Oct 31 00:45:23.441862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.441812 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\") pod \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\" (UID: \"0ccc22fa-fa78-4af3-8413-02e72fdf1bde\") " Oct 31 00:45:23.442199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.442172 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle" pod "0ccc22fa-fa78-4af3-8413-02e72fdf1bde" (UID: "0ccc22fa-fa78-4af3-8413-02e72fdf1bde"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:23.442280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.442233 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config" pod "0ccc22fa-fa78-4af3-8413-02e72fdf1bde" (UID: "0ccc22fa-fa78-4af3-8413-02e72fdf1bde"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:45:23.444201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.444176 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http" pod "0ccc22fa-fa78-4af3-8413-02e72fdf1bde" (UID: "0ccc22fa-fa78-4af3-8413-02e72fdf1bde"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:23.444322 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.444282 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj" pod "0ccc22fa-fa78-4af3-8413-02e72fdf1bde" (UID: "0ccc22fa-fa78-4af3-8413-02e72fdf1bde"). InnerVolumeSpecName "kube-api-access-tsjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:45:23.444387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.444338 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc" pod "0ccc22fa-fa78-4af3-8413-02e72fdf1bde" (UID: "0ccc22fa-fa78-4af3-8413-02e72fdf1bde"). InnerVolumeSpecName "data-science-usage-logs-lokistack-distributor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:45:23.542913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.542839 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsjnj\" (UniqueName: \"kubernetes.io/projected/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-kube-api-access-tsjnj\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.542913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.542867 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.542913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.542878 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-grpc\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-grpc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.542913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.542889 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.542913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.542900 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-distributor-http\" (UniqueName: \"kubernetes.io/secret/0ccc22fa-fa78-4af3-8413-02e72fdf1bde-data-science-usage-logs-lokistack-distributor-http\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:45:23.674071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.674042 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:45:23.678937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:23.678913 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-distributor-6bf78c4576-v89jt"] Oct 31 00:45:25.128945 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:25.128906 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" path="/var/lib/kubelet/pods/0ccc22fa-fa78-4af3-8413-02e72fdf1bde/volumes" Oct 31 00:45:26.967635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:26.967602 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:29.291655 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:29.291623 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:45:31.153741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:31.153693 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.245935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.245903 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.382020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.381982 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.394584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.394558 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:45:37.394993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.394969 5577 cpu_manager.go:414] "containerMap: removing container" podUID="c29bf08b-ce42-4961-a9bd-f45f6866d341" containerName="loki-compactor" Oct 31 00:45:37.394993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.394993 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c29bf08b-ce42-4961-a9bd-f45f6866d341" containerName="loki-compactor" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395006 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8376747b-83f3-42ac-8065-d225284e6727" containerName="otc-container" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395013 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8376747b-83f3-42ac-8065-d225284e6727" containerName="otc-container" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395031 5577 cpu_manager.go:414] "containerMap: removing container" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395037 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395048 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395055 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395079 5577 cpu_manager.go:414] "containerMap: removing container" podUID="64beb84a-7b51-4890-b49e-3d154836ffc0" containerName="loki-query-frontend" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395084 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64beb84a-7b51-4890-b49e-3d154836ffc0" containerName="loki-query-frontend" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395145 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="c29bf08b-ce42-4961-a9bd-f45f6866d341" containerName="loki-compactor" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395152 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8376747b-83f3-42ac-8065-d225284e6727" containerName="otc-container" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395159 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccc22fa-fa78-4af3-8413-02e72fdf1bde" containerName="loki-distributor" Oct 31 00:45:37.395162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395167 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5fd0611-f013-491d-a0db-d3df5f2c38de" containerName="loki-ingester" Oct 31 00:45:37.395657 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.395180 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="64beb84a-7b51-4890-b49e-3d154836ffc0" containerName="loki-query-frontend" Oct 31 00:45:37.399942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.399923 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.409204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.409178 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:45:37.414915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.414892 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:45:37.418075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.418055 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.429611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.429584 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:45:37.475029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.475001 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:45:37.475029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.475006 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-rhrtx\"" Oct 31 00:45:37.475029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.475006 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:45:37.475029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.475009 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-6rjhb\"" Oct 31 00:45:37.495627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495602 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:45:37.495754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495687 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:45:37.495851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495814 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:45:37.495851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495834 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:45:37.495851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495841 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:45:37.495851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.495839 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:45:37.512565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.512536 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.548816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.548786 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwb9\" (UniqueName: \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.548816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.548816 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.549029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.548842 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.549029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.548901 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.549029 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.548948 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.549169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.549061 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnhzc\" (UniqueName: \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.549169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.549093 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.549169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.549137 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.650188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650155 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.650188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650195 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650212 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650261 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnhzc\" (UniqueName: \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650280 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650320 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:37.650320 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:37.650394 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls podName:ffd87a0f-c9ac-409d-ace0-6900c286d135 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:38.150373314 +0000 UTC m=+2777.142032886 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-pvrkg" (UID: "ffd87a0f-c9ac-409d-ace0-6900c286d135") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:45:37.650436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650423 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwb9\" (UniqueName: \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.650831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650462 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.650898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.650840 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.651072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.651051 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.651205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.651173 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.651205 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.651192 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.653048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.653029 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.665057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.665032 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnhzc\" (UniqueName: \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-xhwqc\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.666256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.666237 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwb9\" (UniqueName: \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:37.706506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.706477 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:37.717422 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.717399 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:45:37.939319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.939291 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:45:37.943580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.943556 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:37.953477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.953455 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:45:37.955116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.955087 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:45:37.958226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.958207 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:45:37.958338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.958289 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:37.968004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.967956 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:45:37.987603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:37.987570 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.000850 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:38.000820 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6dba107_55d3_4bbb_931c_36287b3603a9.slice/crio-1888b79583dfcc1aa9f6720a63464e9b310f4add7ed124b24969b2cfdca2c7a5 WatchSource:0}: Error finding container 1888b79583dfcc1aa9f6720a63464e9b310f4add7ed124b24969b2cfdca2c7a5: Status 404 returned error can't find the container with id 1888b79583dfcc1aa9f6720a63464e9b310f4add7ed124b24969b2cfdca2c7a5 Oct 31 00:45:38.004229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.004179 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:45:38.029776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029746 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029773 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029821 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029831 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029773 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029869 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:45:38.029915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.029873 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:45:38.053985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.053956 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.054113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.054024 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vmg\" (UniqueName: \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.054113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.054051 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.054234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.054157 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.054270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.054252 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.054311 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.054299 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.127845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.127815 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.155690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155663 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.155803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155699 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.155803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155746 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vmg\" (UniqueName: \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.155803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155772 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.155803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155791 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:38.155802 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155812 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:38.155861 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs podName:7345a255-f846-42fa-bb9d-cc4fb6be2309 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:38.655845382 +0000 UTC m=+2777.647504944 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.155959 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.156005 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-588tb\" (UniqueName: \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.156039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.156041 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.156360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.156059 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.156857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.156838 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.158803 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.158780 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.158906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.158825 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pvrkg\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:38.159139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.159121 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.159207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.159159 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.170227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.170207 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vmg\" (UniqueName: \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.257121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.257089 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-588tb\" (UniqueName: \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.257468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.257127 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.257468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.257183 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.257544 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.257475 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.257752 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.257735 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.269512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.269483 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-588tb\" (UniqueName: \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:38.335554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.335474 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:45:38.372209 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.372121 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.422437 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.422403 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerStarted","Data":"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85"} Oct 31 00:45:38.422593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.422506 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerStarted","Data":"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771"} Oct 31 00:45:38.422593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.422554 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerStarted","Data":"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d"} Oct 31 00:45:38.422593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.422584 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerStarted","Data":"1888b79583dfcc1aa9f6720a63464e9b310f4add7ed124b24969b2cfdca2c7a5"} Oct 31 00:45:38.445201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.445145 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" podStartSLOduration=1.445130005 podStartE2EDuration="1.445130005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:38.443965009 +0000 UTC m=+2777.435624593" watchObservedRunningTime="2026-10-31 00:45:38.445130005 +0000 UTC m=+2777.436789591" Oct 31 00:45:38.572372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.572333 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:38.616906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.616877 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:45:38.619518 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:38.619490 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd87a0f_c9ac_409d_ace0_6900c286d135.slice/crio-85dd1b46dfc2819a1ae9584194f310dc667abe2ca1f48d8e2a5fc6d1a5a43d8d WatchSource:0}: Error finding container 85dd1b46dfc2819a1ae9584194f310dc667abe2ca1f48d8e2a5fc6d1a5a43d8d: Status 404 returned error can't find the container with id 85dd1b46dfc2819a1ae9584194f310dc667abe2ca1f48d8e2a5fc6d1a5a43d8d Oct 31 00:45:38.660775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.660750 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:38.660935 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:38.660903 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.661051 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:38.660983 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs podName:7345a255-f846-42fa-bb9d-cc4fb6be2309 nodeName:}" failed. No retries permitted until 2026-10-31 00:45:39.660961959 +0000 UTC m=+2778.652621528 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:45:38.966970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:38.966868 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:39.428551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.428507 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerStarted","Data":"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43"} Oct 31 00:45:39.428965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.428585 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerStarted","Data":"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4"} Oct 31 00:45:39.428965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.428621 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerStarted","Data":"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716"} Oct 31 00:45:39.428965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.428654 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerStarted","Data":"85dd1b46dfc2819a1ae9584194f310dc667abe2ca1f48d8e2a5fc6d1a5a43d8d"} Oct 31 00:45:39.448763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.448690 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" podStartSLOduration=2.448675576 podStartE2EDuration="2.448675576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:39.448052654 +0000 UTC m=+2778.439712238" watchObservedRunningTime="2026-10-31 00:45:39.448675576 +0000 UTC m=+2778.440335169" Oct 31 00:45:39.668411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.668328 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:39.670942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.670909 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:39.757527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:39.757495 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:45:40.034034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.034000 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:45:40.037186 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:40.037156 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7345a255_f846_42fa_bb9d_cc4fb6be2309.slice/crio-fa22c64091f01fd1188f2b3869f28439b43ba7b4c67bbfaa1fce45aadaa030b8 WatchSource:0}: Error finding container fa22c64091f01fd1188f2b3869f28439b43ba7b4c67bbfaa1fce45aadaa030b8: Status 404 returned error can't find the container with id fa22c64091f01fd1188f2b3869f28439b43ba7b4c67bbfaa1fce45aadaa030b8 Oct 31 00:45:40.039382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.039353 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:40.039465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.039410 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:45:40.273747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.273695 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.280861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.280831 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.374468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.374371 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.374652 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.374503 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.378011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.377988 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.433868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.433835 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7345a255-f846-42fa-bb9d-cc4fb6be2309","Type":"ContainerStarted","Data":"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0"} Oct 31 00:45:40.434316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.433887 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7345a255-f846-42fa-bb9d-cc4fb6be2309","Type":"ContainerStarted","Data":"fa22c64091f01fd1188f2b3869f28439b43ba7b4c67bbfaa1fce45aadaa030b8"} Oct 31 00:45:40.454745 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.454674 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.454660643 podStartE2EDuration="3.454660643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:40.453931052 +0000 UTC m=+2779.445590637" watchObservedRunningTime="2026-10-31 00:45:40.454660643 +0000 UTC m=+2779.446320226" Oct 31 00:45:40.470215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.470027 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.490581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.490554 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.669831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.669742 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:40.947031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:40.946948 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:45:40.949837 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:40.949807 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47b9c3c1_1d7c_4803_8c21_d5162648bc9a.slice/crio-98c85bc927483256c4938e9e3b56f1d49cc7e5161134373d4d038a93aeee4f58 WatchSource:0}: Error finding container 98c85bc927483256c4938e9e3b56f1d49cc7e5161134373d4d038a93aeee4f58: Status 404 returned error can't find the container with id 98c85bc927483256c4938e9e3b56f1d49cc7e5161134373d4d038a93aeee4f58 Oct 31 00:45:41.440756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:41.440697 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"47b9c3c1-1d7c-4803-8c21-d5162648bc9a","Type":"ContainerStarted","Data":"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726"} Oct 31 00:45:41.441227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:41.440762 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"47b9c3c1-1d7c-4803-8c21-d5162648bc9a","Type":"ContainerStarted","Data":"98c85bc927483256c4938e9e3b56f1d49cc7e5161134373d4d038a93aeee4f58"} Oct 31 00:45:41.441227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:41.440930 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:41.459123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:41.459060 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=4.459030283 podStartE2EDuration="4.459030283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:41.458736311 +0000 UTC m=+2780.450395878" watchObservedRunningTime="2026-10-31 00:45:41.459030283 +0000 UTC m=+2780.450689884" Oct 31 00:45:43.657239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:43.657205 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:43.876645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:43.876613 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.002282 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.002229 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.238974 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.238941 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:45:44.262396 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.262317 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:45:44.262570 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.262461 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.269603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.269574 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:45:44.269794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.269666 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:45:44.269794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.269751 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-gdvlc\"" Oct 31 00:45:44.270002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.269811 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:45:44.270002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.269980 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:45:44.299656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.299626 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.415021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.414982 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415020 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415096 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dqp8\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415170 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415237 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415296 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.415425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.415358 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.447369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.447330 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:44.516873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516795 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.516873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516837 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.516873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516856 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.516873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516872 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.517163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516897 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.517163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516921 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.517163 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.516990 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dqp8\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.517427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.517384 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.519890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.519862 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.520043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.519873 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.520043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.519989 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.521102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.521082 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.521201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.521159 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.530007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.529982 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dqp8\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.585783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.585756 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:44.873785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:44.872956 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:45:44.875824 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:45:44.875791 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5e76cd_d3d3_415a_b48a_e53c33820757.slice/crio-e92945534e49321b0a7d5d2621930b73b3ff225846f93253bf4c7f69090596e8 WatchSource:0}: Error finding container e92945534e49321b0a7d5d2621930b73b3ff225846f93253bf4c7f69090596e8: Status 404 returned error can't find the container with id e92945534e49321b0a7d5d2621930b73b3ff225846f93253bf4c7f69090596e8 Oct 31 00:45:45.459161 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:45.459123 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerStarted","Data":"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd"} Oct 31 00:45:45.459331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:45.459182 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerStarted","Data":"e92945534e49321b0a7d5d2621930b73b3ff225846f93253bf4c7f69090596e8"} Oct 31 00:45:49.478811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:49.478774 5577 generic.go:356] "Generic (PLEG): container finished" podID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerID="a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd" exitCode=0 Oct 31 00:45:49.479252 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:49.478830 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerDied","Data":"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd"} Oct 31 00:45:50.485702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:50.485617 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerStarted","Data":"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b"} Oct 31 00:45:50.485702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:50.485680 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerStarted","Data":"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5"} Oct 31 00:45:50.486150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:50.485892 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:50.513681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:50.513615 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.929385467 podStartE2EDuration="6.513600428s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584214961s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:50.512095208 +0000 UTC m=+2789.503754793" watchObservedRunningTime="2026-10-31 00:45:50.513600428 +0000 UTC m=+2789.505260012" Oct 31 00:45:52.291947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:52.291910 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.423186 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:52.423146 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:45:52.449202 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:52.449176 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:45:56.492140 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:45:56.492109 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:45:59.296382 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:45:59.296298 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:46:17.342725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:17.342684 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.478635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:17.478595 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:17.877854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:17.877816 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:17.878127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:17.878078 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7345a255-f846-42fa-bb9d-cc4fb6be2309" containerName="otc-container" containerID="cri-o://b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0" gracePeriod=30 Oct 31 00:46:17.995694 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:17.995648 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:18.204730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:18.204616 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:19.138469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.138442 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.191123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191091 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\" (UniqueName: \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.191123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191136 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.191384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191188 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.191384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191227 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.191384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191265 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.191384 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191302 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\") pod \"7345a255-f846-42fa-bb9d-cc4fb6be2309\" (UID: \"7345a255-f846-42fa-bb9d-cc4fb6be2309\") " Oct 31 00:46:19.192371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.191811 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:19.193749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.193676 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.193870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.193784 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.193970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.193946 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "kube-api-access-c2vmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:19.194162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.194138 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.194227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.194180 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert" pod "7345a255-f846-42fa-bb9d-cc4fb6be2309" (UID: "7345a255-f846-42fa-bb9d-cc4fb6be2309"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:19.292368 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292333 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2vmg\" (UniqueName: \"kubernetes.io/projected/7345a255-f846-42fa-bb9d-cc4fb6be2309-kube-api-access-c2vmg\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.292368 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292363 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.292368 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292374 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7345a255-f846-42fa-bb9d-cc4fb6be2309-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.292615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292384 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.292615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292392 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.292615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.292401 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7345a255-f846-42fa-bb9d-cc4fb6be2309-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:19.613412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.613326 5577 generic.go:356] "Generic (PLEG): container finished" podID="7345a255-f846-42fa-bb9d-cc4fb6be2309" containerID="b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0" exitCode=0 Oct 31 00:46:19.613412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.613372 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.613412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.613402 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7345a255-f846-42fa-bb9d-cc4fb6be2309","Type":"ContainerDied","Data":"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0"} Oct 31 00:46:19.613680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.613455 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7345a255-f846-42fa-bb9d-cc4fb6be2309","Type":"ContainerDied","Data":"fa22c64091f01fd1188f2b3869f28439b43ba7b4c67bbfaa1fce45aadaa030b8"} Oct 31 00:46:19.613680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.613476 5577 scope.go:122] "RemoveContainer" containerID="b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0" Oct 31 00:46:19.622558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.622540 5577 scope.go:122] "RemoveContainer" containerID="b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0" Oct 31 00:46:19.622861 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:46:19.622843 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0\": container with ID starting with b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0 not found: ID does not exist" containerID="b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0" Oct 31 00:46:19.622939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.622870 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0"} err="failed to get container status \"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0\": rpc error: code = NotFound desc = could not find container \"b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0\": container with ID starting with b41d64e1c266fb134a3edef2eaa324a9185ca5a99338cb8623275dd1e41dcaf0 not found: ID does not exist" Oct 31 00:46:19.635558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.635532 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:19.640232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.640206 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:19.671266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.671228 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:19.672293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.672262 5577 cpu_manager.go:414] "containerMap: removing container" podUID="7345a255-f846-42fa-bb9d-cc4fb6be2309" containerName="otc-container" Oct 31 00:46:19.672466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.672452 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7345a255-f846-42fa-bb9d-cc4fb6be2309" containerName="otc-container" Oct 31 00:46:19.672774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.672758 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="7345a255-f846-42fa-bb9d-cc4fb6be2309" containerName="otc-container" Oct 31 00:46:19.678623 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.678594 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.686689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.686664 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:19.687533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.687512 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:46:19.687672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.687547 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:46:19.687672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.687558 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:46:19.687835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.687564 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:46:19.688054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.688036 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:46:19.688054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.688046 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:46:19.798241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798200 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.798426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798317 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.798426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798351 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhzd9\" (UniqueName: \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.798504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798438 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.798504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798480 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.798569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.798510 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.899776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899664 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.899776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899741 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.900020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899780 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.900020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899833 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.900020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899893 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.900020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.899918 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhzd9\" (UniqueName: \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.900841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.900801 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.902550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.902526 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.902683 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.902658 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.902910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.902889 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.903057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.903037 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.912184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.912162 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhzd9\" (UniqueName: \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\") pod \"data-science-collector-collector-1\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:19.992266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:19.992231 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:20.268507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.268405 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:20.271186 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:46:20.271151 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d92d8c9_169f_45c9_b01a_2fc0df714d5c.slice/crio-247c2aad6f0d8a5e0401b6e42c19097342fea3809d7671100315d92307b06446 WatchSource:0}: Error finding container 247c2aad6f0d8a5e0401b6e42c19097342fea3809d7671100315d92307b06446: Status 404 returned error can't find the container with id 247c2aad6f0d8a5e0401b6e42c19097342fea3809d7671100315d92307b06446 Oct 31 00:46:20.273437 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.273399 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:20.273548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.273466 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:20.621589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.621501 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5d92d8c9-169f-45c9-b01a-2fc0df714d5c","Type":"ContainerStarted","Data":"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521"} Oct 31 00:46:20.621589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.621555 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5d92d8c9-169f-45c9-b01a-2fc0df714d5c","Type":"ContainerStarted","Data":"247c2aad6f0d8a5e0401b6e42c19097342fea3809d7671100315d92307b06446"} Oct 31 00:46:20.643820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:20.643767 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6437547160000001 podStartE2EDuration="1.643754716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:20.642666781 +0000 UTC m=+2819.634326365" watchObservedRunningTime="2026-10-31 00:46:20.643754716 +0000 UTC m=+2819.635414299" Oct 31 00:46:21.131465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:21.131425 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7345a255-f846-42fa-bb9d-cc4fb6be2309" path="/var/lib/kubelet/pods/7345a255-f846-42fa-bb9d-cc4fb6be2309/volumes" Oct 31 00:46:21.764472 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:21.764432 5577 scope.go:122] "RemoveContainer" containerID="d08089ae055a228ffc859eab34c08c8de623c26e7112952da433f00fc6788e2a" Oct 31 00:46:27.023236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:27.023199 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.291690 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:46:29.291650 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:46:29.444618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:29.444586 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.576031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:29.575936 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:29.969676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:29.969582 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:29.969880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:29.969829 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" containerName="otc-container" containerID="cri-o://eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521" gracePeriod=30 Oct 31 00:46:30.102121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.102087 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.217838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.217811 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.226969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.226901 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:46:30.299234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299104 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299147 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299291 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299343 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\" (UniqueName: \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299395 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299431 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\") pod \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\" (UID: \"5d92d8c9-169f-45c9-b01a-2fc0df714d5c\") " Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299503 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:46:30.299820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.299682 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.301639 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.301611 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:30.301773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.301634 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:30.301773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.301667 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "kube-api-access-mhzd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:46:30.301990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.301969 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:30.302059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.302043 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert" pod "5d92d8c9-169f-45c9-b01a-2fc0df714d5c" (UID: "5d92d8c9-169f-45c9-b01a-2fc0df714d5c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:46:30.400822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.400787 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.400822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.400823 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.400822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.400834 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.401167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.400843 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.401167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.400854 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhzd9\" (UniqueName: \"kubernetes.io/projected/5d92d8c9-169f-45c9-b01a-2fc0df714d5c-kube-api-access-mhzd9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:46:30.670600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.670554 5577 generic.go:356] "Generic (PLEG): container finished" podID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" containerID="eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521" exitCode=0 Oct 31 00:46:30.670836 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.670680 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5d92d8c9-169f-45c9-b01a-2fc0df714d5c","Type":"ContainerDied","Data":"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521"} Oct 31 00:46:30.670836 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.670759 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5d92d8c9-169f-45c9-b01a-2fc0df714d5c","Type":"ContainerDied","Data":"247c2aad6f0d8a5e0401b6e42c19097342fea3809d7671100315d92307b06446"} Oct 31 00:46:30.670836 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.670792 5577 scope.go:122] "RemoveContainer" containerID="eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521" Oct 31 00:46:30.671023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.670970 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.681152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.681113 5577 scope.go:122] "RemoveContainer" containerID="eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521" Oct 31 00:46:30.681518 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:46:30.681449 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521\": container with ID starting with eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521 not found: ID does not exist" containerID="eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521" Oct 31 00:46:30.681518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.681489 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521"} err="failed to get container status \"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521\": rpc error: code = NotFound desc = could not find container \"eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521\": container with ID starting with eb3b5f07a046291d60f9cf69e3ba5d94b86dde0a6b4881a28c3ae82d2f49e521 not found: ID does not exist" Oct 31 00:46:30.696356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.696332 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:30.703084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.703061 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:30.731551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.731527 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:30.731891 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.731878 5577 cpu_manager.go:414] "containerMap: removing container" podUID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" containerName="otc-container" Oct 31 00:46:30.731953 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.731893 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" containerName="otc-container" Oct 31 00:46:30.732000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.731968 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" containerName="otc-container" Oct 31 00:46:30.736431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.736416 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.740910 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.740889 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:46:30.741034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.740976 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:46:30.741034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.740990 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:46:30.741034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.741005 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:46:30.741034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.741027 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:46:30.746330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.746307 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:30.770317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.770286 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:46:30.803723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803678 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.803819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803725 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.803819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803755 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.803913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803838 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kltm9\" (UniqueName: \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.803913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803878 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.804025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.803930 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.904978 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905017 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905042 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905062 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kltm9\" (UniqueName: \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905080 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905117 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.905906 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.905877 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.908122 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.908092 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.908251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.908231 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.908327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.908273 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.908631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.908612 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:30.918105 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:30.918084 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kltm9\" (UniqueName: \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\") pod \"data-science-collector-collector-1\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:31.050647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.050613 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:46:31.130169 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.130134 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d92d8c9-169f-45c9-b01a-2fc0df714d5c" path="/var/lib/kubelet/pods/5d92d8c9-169f-45c9-b01a-2fc0df714d5c/volumes" Oct 31 00:46:31.336423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.336398 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:46:31.339052 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:46:31.339019 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7681f267_6a25_47d4_8808_95c279de1c41.slice/crio-7a927c45c964d0cd86b89c42347315466b90bf04c0d33f6cd84d4cbb8a40c463 WatchSource:0}: Error finding container 7a927c45c964d0cd86b89c42347315466b90bf04c0d33f6cd84d4cbb8a40c463: Status 404 returned error can't find the container with id 7a927c45c964d0cd86b89c42347315466b90bf04c0d33f6cd84d4cbb8a40c463 Oct 31 00:46:31.341195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.341164 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:31.341283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.341221 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:46:31.677391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.677297 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7681f267-6a25-47d4-8808-95c279de1c41","Type":"ContainerStarted","Data":"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813"} Oct 31 00:46:31.677391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.677357 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7681f267-6a25-47d4-8808-95c279de1c41","Type":"ContainerStarted","Data":"7a927c45c964d0cd86b89c42347315466b90bf04c0d33f6cd84d4cbb8a40c463"} Oct 31 00:46:31.700788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:46:31.700729 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.700694104 podStartE2EDuration="1.700694104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:31.699431204 +0000 UTC m=+2830.691090790" watchObservedRunningTime="2026-10-31 00:46:31.700694104 +0000 UTC m=+2830.692353689" Oct 31 00:46:59.292318 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:46:59.292283 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:47:03.136631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.136595 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.271691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.271650 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.676074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.676038 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:03.676325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.676295 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7681f267-6a25-47d4-8808-95c279de1c41" containerName="otc-container" containerID="cri-o://d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813" gracePeriod=30 Oct 31 00:47:03.785063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.785026 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:03.916922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:03.916887 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:06.033234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.033174 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:06.089613 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089576 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.089799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089625 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\" (UniqueName: \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.089799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089664 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.089799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089694 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.089799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089772 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.090025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.089819 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\") pod \"7681f267-6a25-47d4-8808-95c279de1c41\" (UID: \"7681f267-6a25-47d4-8808-95c279de1c41\") " Oct 31 00:47:06.090236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.090201 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:06.092007 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.091968 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.092431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.092410 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "kube-api-access-kltm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:06.092503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.092411 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.092612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.092589 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.092653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.092601 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert" pod "7681f267-6a25-47d4-8808-95c279de1c41" (UID: "7681f267-6a25-47d4-8808-95c279de1c41"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:06.191361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191324 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.191361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191360 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kltm9\" (UniqueName: \"kubernetes.io/projected/7681f267-6a25-47d4-8808-95c279de1c41-kube-api-access-kltm9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.191549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191375 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.191549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191390 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.191549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191404 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7681f267-6a25-47d4-8808-95c279de1c41-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.191549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.191418 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7681f267-6a25-47d4-8808-95c279de1c41-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:06.827539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.827508 5577 generic.go:356] "Generic (PLEG): container finished" podID="7681f267-6a25-47d4-8808-95c279de1c41" containerID="d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813" exitCode=0 Oct 31 00:47:06.827759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.827552 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:06.827759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.827601 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7681f267-6a25-47d4-8808-95c279de1c41","Type":"ContainerDied","Data":"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813"} Oct 31 00:47:06.827759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.827651 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7681f267-6a25-47d4-8808-95c279de1c41","Type":"ContainerDied","Data":"7a927c45c964d0cd86b89c42347315466b90bf04c0d33f6cd84d4cbb8a40c463"} Oct 31 00:47:06.827759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.827675 5577 scope.go:122] "RemoveContainer" containerID="d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813" Oct 31 00:47:06.836171 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.836157 5577 scope.go:122] "RemoveContainer" containerID="d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813" Oct 31 00:47:06.836414 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:06.836397 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813\": container with ID starting with d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813 not found: ID does not exist" containerID="d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813" Oct 31 00:47:06.836461 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.836420 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813"} err="failed to get container status \"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813\": rpc error: code = NotFound desc = could not find container \"d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813\": container with ID starting with d4c1a2507ad0a9d221a3cb588c9bfb25bb2362e81d67f20c97e7486ccea75813 not found: ID does not exist" Oct 31 00:47:06.848985 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.848966 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:06.853573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.853554 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:06.882372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.882342 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:06.882691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.882677 5577 cpu_manager.go:414] "containerMap: removing container" podUID="7681f267-6a25-47d4-8808-95c279de1c41" containerName="otc-container" Oct 31 00:47:06.882788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.882691 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7681f267-6a25-47d4-8808-95c279de1c41" containerName="otc-container" Oct 31 00:47:06.882788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.882772 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="7681f267-6a25-47d4-8808-95c279de1c41" containerName="otc-container" Oct 31 00:47:06.887054 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.887034 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:06.891419 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.891390 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:47:06.891540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.891424 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:47:06.891540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.891453 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:47:06.891641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.891566 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:47:06.891641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.891564 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-7hrqp\"" Oct 31 00:47:06.892479 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.892462 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:47:06.903046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:06.903023 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:07.002184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002147 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.002359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002194 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6l9q\" (UniqueName: \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.002359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002269 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.002359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002324 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.002359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002349 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.002507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.002439 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.103759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103649 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.103759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103749 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.104299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103774 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6l9q\" (UniqueName: \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.104299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103794 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.104299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103845 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.104299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.103876 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.104642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.104481 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.106557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.106520 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.106676 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.106587 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.106963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.106943 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.107025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.107009 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.116552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.116534 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6l9q\" (UniqueName: \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\") pod \"data-science-collector-collector-1\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.129660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.129635 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7681f267-6a25-47d4-8808-95c279de1c41" path="/var/lib/kubelet/pods/7681f267-6a25-47d4-8808-95c279de1c41/volumes" Oct 31 00:47:07.206099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.206071 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:07.480929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.480905 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:07.483160 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:07.483131 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod336f26a3_2824_4671_8109_a023535f223c.slice/crio-9be99353907d5f37533ab34d8eb955dd25a8525c6df7fc20c3f756ec72611c69 WatchSource:0}: Error finding container 9be99353907d5f37533ab34d8eb955dd25a8525c6df7fc20c3f756ec72611c69: Status 404 returned error can't find the container with id 9be99353907d5f37533ab34d8eb955dd25a8525c6df7fc20c3f756ec72611c69 Oct 31 00:47:07.485414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.485377 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:07.485519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.485456 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:07.834234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.834195 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"336f26a3-2824-4671-8109-a023535f223c","Type":"ContainerStarted","Data":"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508"} Oct 31 00:47:07.834395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.834256 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"336f26a3-2824-4671-8109-a023535f223c","Type":"ContainerStarted","Data":"9be99353907d5f37533ab34d8eb955dd25a8525c6df7fc20c3f756ec72611c69"} Oct 31 00:47:07.855843 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:07.855764 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.855749377 podStartE2EDuration="1.855749377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:07.854872171 +0000 UTC m=+2866.846531757" watchObservedRunningTime="2026-10-31 00:47:07.855749377 +0000 UTC m=+2866.847408962" Oct 31 00:47:09.283562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.283528 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:47:09.284078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.283772 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="kube-rbac-proxy" containerID="cri-o://e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" gracePeriod=30 Oct 31 00:47:09.284078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.283866 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="method-gate" containerID="cri-o://0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" gracePeriod=30 Oct 31 00:47:09.284078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.283848 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="prom-label-proxy" containerID="cri-o://368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" gracePeriod=30 Oct 31 00:47:09.288659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.288630 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:47:09.289093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.288895 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="kube-rbac-proxy" containerID="cri-o://b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" gracePeriod=30 Oct 31 00:47:09.289093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.288923 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="prom-label-proxy" containerID="cri-o://7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" gracePeriod=30 Oct 31 00:47:09.289093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.289028 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="method-gate" containerID="cri-o://0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" gracePeriod=30 Oct 31 00:47:09.494239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.493614 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.511753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.510782 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:09.511753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.511028 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="alertmanager" containerID="cri-o://7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" gracePeriod=120 Oct 31 00:47:09.511753 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.511080 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="config-reloader" containerID="cri-o://410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" gracePeriod=120 Oct 31 00:47:09.585590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.585566 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:47:09.588575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.588556 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:47:09.626392 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626360 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\" (UniqueName: \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\") pod \"e6dba107-55d3-4bbb-931c-36287b3603a9\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " Oct 31 00:47:09.626592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626409 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\" (UniqueName: \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\") pod \"ffd87a0f-c9ac-409d-ace0-6900c286d135\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " Oct 31 00:47:09.626592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626469 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\") pod \"e6dba107-55d3-4bbb-931c-36287b3603a9\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " Oct 31 00:47:09.626592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626528 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") pod \"ffd87a0f-c9ac-409d-ace0-6900c286d135\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " Oct 31 00:47:09.626592 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626571 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\") pod \"e6dba107-55d3-4bbb-931c-36287b3603a9\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " Oct 31 00:47:09.626846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626612 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\" (UniqueName: \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\") pod \"e6dba107-55d3-4bbb-931c-36287b3603a9\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " Oct 31 00:47:09.626846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626667 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\") pod \"e6dba107-55d3-4bbb-931c-36287b3603a9\" (UID: \"e6dba107-55d3-4bbb-931c-36287b3603a9\") " Oct 31 00:47:09.626846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626703 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\" (UniqueName: \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\") pod \"ffd87a0f-c9ac-409d-ace0-6900c286d135\" (UID: \"ffd87a0f-c9ac-409d-ace0-6900c286d135\") " Oct 31 00:47:09.627012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.626860 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config" pod "ffd87a0f-c9ac-409d-ace0-6900c286d135" (UID: "ffd87a0f-c9ac-409d-ace0-6900c286d135"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.627129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.627105 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca" pod "e6dba107-55d3-4bbb-931c-36287b3603a9" (UID: "e6dba107-55d3-4bbb-931c-36287b3603a9"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.627197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.627138 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffd87a0f-c9ac-409d-ace0-6900c286d135-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.627197 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.627175 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca" pod "e6dba107-55d3-4bbb-931c-36287b3603a9" (UID: "e6dba107-55d3-4bbb-931c-36287b3603a9"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.627473 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.627450 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config" pod "e6dba107-55d3-4bbb-931c-36287b3603a9" (UID: "e6dba107-55d3-4bbb-931c-36287b3603a9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:09.629231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.629203 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls" pod "ffd87a0f-c9ac-409d-ace0-6900c286d135" (UID: "ffd87a0f-c9ac-409d-ace0-6900c286d135"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.629331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.629243 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls" pod "e6dba107-55d3-4bbb-931c-36287b3603a9" (UID: "e6dba107-55d3-4bbb-931c-36287b3603a9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:09.629331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.629264 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9" pod "ffd87a0f-c9ac-409d-ace0-6900c286d135" (UID: "ffd87a0f-c9ac-409d-ace0-6900c286d135"). InnerVolumeSpecName "kube-api-access-zbwb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:09.629468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.629453 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc" pod "e6dba107-55d3-4bbb-931c-36287b3603a9" (UID: "e6dba107-55d3-4bbb-931c-36287b3603a9"). InnerVolumeSpecName "kube-api-access-vnhzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:09.637027 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.636279 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.657868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.657835 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:09.727890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727858 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffd87a0f-c9ac-409d-ace0-6900c286d135-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.727890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727890 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.728125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727905 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnhzc\" (UniqueName: \"kubernetes.io/projected/e6dba107-55d3-4bbb-931c-36287b3603a9-kube-api-access-vnhzc\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.728125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727920 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.728125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727933 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbwb9\" (UniqueName: \"kubernetes.io/projected/ffd87a0f-c9ac-409d-ace0-6900c286d135-kube-api-access-zbwb9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.728125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727944 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/e6dba107-55d3-4bbb-931c-36287b3603a9-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.728125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.727956 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/e6dba107-55d3-4bbb-931c-36287b3603a9-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:09.742837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.742805 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:09.743026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.742983 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" containerName="perses" containerID="cri-o://1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726" gracePeriod=30 Oct 31 00:47:09.781818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.781779 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.845492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845403 5577 generic.go:356] "Generic (PLEG): container finished" podID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" exitCode=0 Oct 31 00:47:09.845492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845432 5577 generic.go:356] "Generic (PLEG): container finished" podID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" exitCode=0 Oct 31 00:47:09.845492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845440 5577 generic.go:356] "Generic (PLEG): container finished" podID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" exitCode=0 Oct 31 00:47:09.845492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845474 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" Oct 31 00:47:09.845492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845484 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerDied","Data":"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85"} Oct 31 00:47:09.845898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845551 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerDied","Data":"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771"} Oct 31 00:47:09.845898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845591 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerDied","Data":"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d"} Oct 31 00:47:09.845898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845619 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc" event={"ID":"e6dba107-55d3-4bbb-931c-36287b3603a9","Type":"ContainerDied","Data":"1888b79583dfcc1aa9f6720a63464e9b310f4add7ed124b24969b2cfdca2c7a5"} Oct 31 00:47:09.845898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.845656 5577 scope.go:122] "RemoveContainer" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" Oct 31 00:47:09.853749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.849980 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" exitCode=0 Oct 31 00:47:09.853749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.850007 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" exitCode=0 Oct 31 00:47:09.853749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.850044 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" exitCode=0 Oct 31 00:47:09.853749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.850294 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="336f26a3-2824-4671-8109-a023535f223c" containerName="otc-container" containerID="cri-o://762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508" gracePeriod=30 Oct 31 00:47:09.853749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.850463 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" Oct 31 00:47:09.854748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.854671 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerDied","Data":"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43"} Oct 31 00:47:09.854886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.854768 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerDied","Data":"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4"} Oct 31 00:47:09.854886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.854806 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerDied","Data":"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716"} Oct 31 00:47:09.854886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.854842 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg" event={"ID":"ffd87a0f-c9ac-409d-ace0-6900c286d135","Type":"ContainerDied","Data":"85dd1b46dfc2819a1ae9584194f310dc667abe2ca1f48d8e2a5fc6d1a5a43d8d"} Oct 31 00:47:09.875546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.875514 5577 scope.go:122] "RemoveContainer" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" Oct 31 00:47:09.903583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.903544 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:47:09.907991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.907960 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pvrkg"] Oct 31 00:47:09.926831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.925378 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:47:09.927304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.927283 5577 scope.go:122] "RemoveContainer" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" Oct 31 00:47:09.930821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.929648 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:09.931850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.931796 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-xhwqc"] Oct 31 00:47:09.973933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.973805 5577 scope.go:122] "RemoveContainer" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" Oct 31 00:47:09.974678 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:09.974650 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": container with ID starting with 7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85 not found: ID does not exist" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" Oct 31 00:47:09.974826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.974687 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85"} err="failed to get container status \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": rpc error: code = NotFound desc = could not find container \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": container with ID starting with 7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85 not found: ID does not exist" Oct 31 00:47:09.974826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.974737 5577 scope.go:122] "RemoveContainer" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" Oct 31 00:47:09.975117 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:09.975091 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": container with ID starting with 0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771 not found: ID does not exist" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" Oct 31 00:47:09.975117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.975128 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771"} err="failed to get container status \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": rpc error: code = NotFound desc = could not find container \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": container with ID starting with 0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771 not found: ID does not exist" Oct 31 00:47:09.975117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.975149 5577 scope.go:122] "RemoveContainer" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" Oct 31 00:47:09.975785 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:09.975752 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": container with ID starting with b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d not found: ID does not exist" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" Oct 31 00:47:09.975879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.975790 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d"} err="failed to get container status \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": rpc error: code = NotFound desc = could not find container \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": container with ID starting with b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d not found: ID does not exist" Oct 31 00:47:09.975879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.975830 5577 scope.go:122] "RemoveContainer" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" Oct 31 00:47:09.976160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976137 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85"} err="failed to get container status \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": rpc error: code = NotFound desc = could not find container \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": container with ID starting with 7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85 not found: ID does not exist" Oct 31 00:47:09.976160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976159 5577 scope.go:122] "RemoveContainer" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" Oct 31 00:47:09.976443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976419 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771"} err="failed to get container status \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": rpc error: code = NotFound desc = could not find container \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": container with ID starting with 0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771 not found: ID does not exist" Oct 31 00:47:09.976443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976444 5577 scope.go:122] "RemoveContainer" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" Oct 31 00:47:09.976762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976737 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d"} err="failed to get container status \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": rpc error: code = NotFound desc = could not find container \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": container with ID starting with b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d not found: ID does not exist" Oct 31 00:47:09.976762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976760 5577 scope.go:122] "RemoveContainer" containerID="7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85" Oct 31 00:47:09.977017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.976991 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85"} err="failed to get container status \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": rpc error: code = NotFound desc = could not find container \"7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85\": container with ID starting with 7fbc7ae73b0802a0bb234ac459ad58fa5b692758ec971a22d1408a44ccb34d85 not found: ID does not exist" Oct 31 00:47:09.977135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.977017 5577 scope.go:122] "RemoveContainer" containerID="0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771" Oct 31 00:47:09.978316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.977327 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771"} err="failed to get container status \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": rpc error: code = NotFound desc = could not find container \"0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771\": container with ID starting with 0f66ee5ce2995c51dfff1b85d35a4a458d18118671233f282f156be1c271f771 not found: ID does not exist" Oct 31 00:47:09.978316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.977358 5577 scope.go:122] "RemoveContainer" containerID="b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d" Oct 31 00:47:09.978316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.977590 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d"} err="failed to get container status \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": rpc error: code = NotFound desc = could not find container \"b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d\": container with ID starting with b31d826b099c197986a9ac16b3a4496c2797a8da171c40bc2bf7c614a448e57d not found: ID does not exist" Oct 31 00:47:09.978316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.977612 5577 scope.go:122] "RemoveContainer" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" Oct 31 00:47:09.992952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:09.992831 5577 scope.go:122] "RemoveContainer" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" Oct 31 00:47:10.003616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.003597 5577 scope.go:122] "RemoveContainer" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" Oct 31 00:47:10.016874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.016851 5577 scope.go:122] "RemoveContainer" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" Oct 31 00:47:10.017497 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.017471 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": container with ID starting with 368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43 not found: ID does not exist" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" Oct 31 00:47:10.017642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.017506 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43"} err="failed to get container status \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": rpc error: code = NotFound desc = could not find container \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": container with ID starting with 368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43 not found: ID does not exist" Oct 31 00:47:10.017642 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.017525 5577 scope.go:122] "RemoveContainer" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" Oct 31 00:47:10.017952 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.017899 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": container with ID starting with 0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4 not found: ID does not exist" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" Oct 31 00:47:10.017952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.017925 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4"} err="failed to get container status \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": rpc error: code = NotFound desc = could not find container \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": container with ID starting with 0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4 not found: ID does not exist" Oct 31 00:47:10.017952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.017941 5577 scope.go:122] "RemoveContainer" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" Oct 31 00:47:10.018269 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.018248 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": container with ID starting with e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716 not found: ID does not exist" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" Oct 31 00:47:10.018337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.018277 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716"} err="failed to get container status \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": rpc error: code = NotFound desc = could not find container \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": container with ID starting with e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716 not found: ID does not exist" Oct 31 00:47:10.018337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.018293 5577 scope.go:122] "RemoveContainer" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" Oct 31 00:47:10.026603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.025934 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43"} err="failed to get container status \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": rpc error: code = NotFound desc = could not find container \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": container with ID starting with 368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43 not found: ID does not exist" Oct 31 00:47:10.026603 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.025973 5577 scope.go:122] "RemoveContainer" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" Oct 31 00:47:10.026995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.026965 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4"} err="failed to get container status \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": rpc error: code = NotFound desc = could not find container \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": container with ID starting with 0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4 not found: ID does not exist" Oct 31 00:47:10.026995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.026993 5577 scope.go:122] "RemoveContainer" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" Oct 31 00:47:10.027357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.027323 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716"} err="failed to get container status \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": rpc error: code = NotFound desc = could not find container \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": container with ID starting with e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716 not found: ID does not exist" Oct 31 00:47:10.027424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.027362 5577 scope.go:122] "RemoveContainer" containerID="368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43" Oct 31 00:47:10.027681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.027649 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43"} err="failed to get container status \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": rpc error: code = NotFound desc = could not find container \"368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43\": container with ID starting with 368b3a09ad039a24acc81037047be1a4eff5281fef4f50594695d2831bfc7f43 not found: ID does not exist" Oct 31 00:47:10.027681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.027675 5577 scope.go:122] "RemoveContainer" containerID="0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4" Oct 31 00:47:10.029220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.029189 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4"} err="failed to get container status \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": rpc error: code = NotFound desc = could not find container \"0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4\": container with ID starting with 0d36d03a4ae32e31e946e849bffbb21f4f6a4f5041152700e6ad52ae960b3fd4 not found: ID does not exist" Oct 31 00:47:10.029556 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.029541 5577 scope.go:122] "RemoveContainer" containerID="e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716" Oct 31 00:47:10.030305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.030274 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716"} err="failed to get container status \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": rpc error: code = NotFound desc = could not find container \"e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716\": container with ID starting with e5f3594541f908f2051adc7bd74856749e4d7e5d991b50d784aacf1849f0a716 not found: ID does not exist" Oct 31 00:47:10.067595 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.067561 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:10.068124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.068104 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:10.118810 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.118785 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:10.131501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.131479 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\" (UniqueName: \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\") pod \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " Oct 31 00:47:10.131681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.131564 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\" (UniqueName: \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\") pod \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " Oct 31 00:47:10.131681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.131672 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " Oct 31 00:47:10.131829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.131777 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\" (UniqueName: \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\") pod \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\" (UID: \"47b9c3c1-1d7c-4803-8c21-d5162648bc9a\") " Oct 31 00:47:10.131939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.131916 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config" pod "47b9c3c1-1d7c-4803-8c21-d5162648bc9a" (UID: "47b9c3c1-1d7c-4803-8c21-d5162648bc9a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.132045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.132027 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.134491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.134471 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "47b9c3c1-1d7c-4803-8c21-d5162648bc9a" (UID: "47b9c3c1-1d7c-4803-8c21-d5162648bc9a"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:47:10.134605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.134507 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb" pod "47b9c3c1-1d7c-4803-8c21-d5162648bc9a" (UID: "47b9c3c1-1d7c-4803-8c21-d5162648bc9a"). InnerVolumeSpecName "kube-api-access-588tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.233170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233137 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233170 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233179 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233213 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233237 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233258 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233283 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\" (UniqueName: \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\") pod \"336f26a3-2824-4671-8109-a023535f223c\" (UID: \"336f26a3-2824-4671-8109-a023535f223c\") " Oct 31 00:47:10.233646 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233622 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:47:10.233739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233646 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-588tb\" (UniqueName: \"kubernetes.io/projected/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-kube-api-access-588tb\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.233739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.233681 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:10.236576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.236543 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.236738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.236549 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "kube-api-access-l6l9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.236738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.236600 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.236738 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.236636 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.238625 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.238590 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs" pod "336f26a3-2824-4671-8109-a023535f223c" (UID: "336f26a3-2824-4671-8109-a023535f223c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.248555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.248533 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:47:10.290854 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.290815 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins" pod "47b9c3c1-1d7c-4803-8c21-d5162648bc9a" (UID: "47b9c3c1-1d7c-4803-8c21-d5162648bc9a"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:10.334427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334393 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/47b9c3c1-1d7c-4803-8c21-d5162648bc9a-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334427 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334423 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334434 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334446 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334457 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334465 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/336f26a3-2824-4671-8109-a023535f223c-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334474 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/336f26a3-2824-4671-8109-a023535f223c-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.334617 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.334482 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6l9q\" (UniqueName: \"kubernetes.io/projected/336f26a3-2824-4671-8109-a023535f223c-kube-api-access-l6l9q\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.371735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.371658 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:10.435333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435303 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435368 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435389 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435510 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435571 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435619 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435695 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\") pod \"cf5e76cd-d3d3-415a-b48a-e53c33820757\" (UID: \"cf5e76cd-d3d3-415a-b48a-e53c33820757\") " Oct 31 00:47:10.435989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.435945 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:10.437816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.437790 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.438198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.438177 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.438295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.438278 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "kube-api-access-7dqp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:10.438358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.438292 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:47:10.441886 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.441861 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.448826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.448692 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config" pod "cf5e76cd-d3d3-415a-b48a-e53c33820757" (UID: "cf5e76cd-d3d3-415a-b48a-e53c33820757"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:10.536523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536492 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536517 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536527 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536535 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/cf5e76cd-d3d3-415a-b48a-e53c33820757-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536546 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536555 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dqp8\" (UniqueName: \"kubernetes.io/projected/cf5e76cd-d3d3-415a-b48a-e53c33820757-kube-api-access-7dqp8\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.536822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.536564 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf5e76cd-d3d3-415a-b48a-e53c33820757-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:10.858573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.858541 5577 generic.go:356] "Generic (PLEG): container finished" podID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" containerID="1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726" exitCode=0 Oct 31 00:47:10.858763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.858583 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:10.858763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.858632 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"47b9c3c1-1d7c-4803-8c21-d5162648bc9a","Type":"ContainerDied","Data":"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726"} Oct 31 00:47:10.858763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.858677 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"47b9c3c1-1d7c-4803-8c21-d5162648bc9a","Type":"ContainerDied","Data":"98c85bc927483256c4938e9e3b56f1d49cc7e5161134373d4d038a93aeee4f58"} Oct 31 00:47:10.858763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.858703 5577 scope.go:122] "RemoveContainer" containerID="1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726" Oct 31 00:47:10.860101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.860079 5577 generic.go:356] "Generic (PLEG): container finished" podID="336f26a3-2824-4671-8109-a023535f223c" containerID="762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508" exitCode=0 Oct 31 00:47:10.860203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.860132 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:10.860203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.860149 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"336f26a3-2824-4671-8109-a023535f223c","Type":"ContainerDied","Data":"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508"} Oct 31 00:47:10.860287 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.860196 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"336f26a3-2824-4671-8109-a023535f223c","Type":"ContainerDied","Data":"9be99353907d5f37533ab34d8eb955dd25a8525c6df7fc20c3f756ec72611c69"} Oct 31 00:47:10.862210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862190 5577 generic.go:356] "Generic (PLEG): container finished" podID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerID="410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" exitCode=0 Oct 31 00:47:10.862210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862209 5577 generic.go:356] "Generic (PLEG): container finished" podID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerID="7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" exitCode=0 Oct 31 00:47:10.862338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862238 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerDied","Data":"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b"} Oct 31 00:47:10.862338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862270 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerDied","Data":"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5"} Oct 31 00:47:10.862338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862293 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"cf5e76cd-d3d3-415a-b48a-e53c33820757","Type":"ContainerDied","Data":"e92945534e49321b0a7d5d2621930b73b3ff225846f93253bf4c7f69090596e8"} Oct 31 00:47:10.862338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.862268 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:10.868461 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.868443 5577 scope.go:122] "RemoveContainer" containerID="1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726" Oct 31 00:47:10.868843 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.868811 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726\": container with ID starting with 1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726 not found: ID does not exist" containerID="1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726" Oct 31 00:47:10.868975 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.868849 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726"} err="failed to get container status \"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726\": rpc error: code = NotFound desc = could not find container \"1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726\": container with ID starting with 1c80a64be4e4e8982b1113749b2c579fb151c66ae50d9a1b989b162458477726 not found: ID does not exist" Oct 31 00:47:10.868975 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.868873 5577 scope.go:122] "RemoveContainer" containerID="762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508" Oct 31 00:47:10.877370 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.877350 5577 scope.go:122] "RemoveContainer" containerID="762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508" Oct 31 00:47:10.877623 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.877605 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508\": container with ID starting with 762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508 not found: ID does not exist" containerID="762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508" Oct 31 00:47:10.877674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.877631 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508"} err="failed to get container status \"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508\": rpc error: code = NotFound desc = could not find container \"762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508\": container with ID starting with 762f70be125c9b75038ebe50fcde121c7ee0e9944b3d04d601f2ec923304b508 not found: ID does not exist" Oct 31 00:47:10.877674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.877647 5577 scope.go:122] "RemoveContainer" containerID="410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" Oct 31 00:47:10.885213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.885195 5577 scope.go:122] "RemoveContainer" containerID="7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" Oct 31 00:47:10.888775 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.888755 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:10.893645 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.893624 5577 scope.go:122] "RemoveContainer" containerID="a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd" Oct 31 00:47:10.894276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.894251 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:10.900959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.900935 5577 scope.go:122] "RemoveContainer" containerID="410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" Oct 31 00:47:10.901230 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.901206 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b\": container with ID starting with 410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b not found: ID does not exist" containerID="410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" Oct 31 00:47:10.901289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901239 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b"} err="failed to get container status \"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b\": rpc error: code = NotFound desc = could not find container \"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b\": container with ID starting with 410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b not found: ID does not exist" Oct 31 00:47:10.901289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901256 5577 scope.go:122] "RemoveContainer" containerID="7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" Oct 31 00:47:10.901501 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.901482 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5\": container with ID starting with 7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5 not found: ID does not exist" containerID="7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" Oct 31 00:47:10.901539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901509 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5"} err="failed to get container status \"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5\": rpc error: code = NotFound desc = could not find container \"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5\": container with ID starting with 7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5 not found: ID does not exist" Oct 31 00:47:10.901539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901524 5577 scope.go:122] "RemoveContainer" containerID="a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd" Oct 31 00:47:10.901754 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:10.901737 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd\": container with ID starting with a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd not found: ID does not exist" containerID="a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd" Oct 31 00:47:10.901821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901757 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd"} err="failed to get container status \"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd\": rpc error: code = NotFound desc = could not find container \"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd\": container with ID starting with a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd not found: ID does not exist" Oct 31 00:47:10.901821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901770 5577 scope.go:122] "RemoveContainer" containerID="410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b" Oct 31 00:47:10.902002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.901982 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b"} err="failed to get container status \"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b\": rpc error: code = NotFound desc = could not find container \"410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b\": container with ID starting with 410b86ea61fa2e2eca96074041bf96846ec2077007bd740c2eb700b7c66a4f2b not found: ID does not exist" Oct 31 00:47:10.902068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.902005 5577 scope.go:122] "RemoveContainer" containerID="7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5" Oct 31 00:47:10.902229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.902214 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5"} err="failed to get container status \"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5\": rpc error: code = NotFound desc = could not find container \"7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5\": container with ID starting with 7e2d4336ed4655a33d90ca661b7f67e5ffe1144b0faa657ad762dfcc11b2b0d5 not found: ID does not exist" Oct 31 00:47:10.902278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.902229 5577 scope.go:122] "RemoveContainer" containerID="a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd" Oct 31 00:47:10.902450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.902431 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd"} err="failed to get container status \"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd\": rpc error: code = NotFound desc = could not find container \"a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd\": container with ID starting with a02815fe498c316903774064d842c440ec569a24a9a1576ad4f2f1416c8c2dbd not found: ID does not exist" Oct 31 00:47:10.907128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.907106 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:10.912307 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.912288 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:10.926327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.926301 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:10.933344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:10.933321 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:11.129928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:11.129836 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="336f26a3-2824-4671-8109-a023535f223c" path="/var/lib/kubelet/pods/336f26a3-2824-4671-8109-a023535f223c/volumes" Oct 31 00:47:11.130339 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:11.130319 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" path="/var/lib/kubelet/pods/47b9c3c1-1d7c-4803-8c21-d5162648bc9a/volumes" Oct 31 00:47:11.130746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:11.130727 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" path="/var/lib/kubelet/pods/cf5e76cd-d3d3-415a-b48a-e53c33820757/volumes" Oct 31 00:47:11.131672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:11.131651 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" path="/var/lib/kubelet/pods/e6dba107-55d3-4bbb-931c-36287b3603a9/volumes" Oct 31 00:47:11.132153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:11.132135 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" path="/var/lib/kubelet/pods/ffd87a0f-c9ac-409d-ace0-6900c286d135/volumes" Oct 31 00:47:17.543156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.543117 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.613794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.613765 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:47:17.614048 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614037 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="method-gate" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614049 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="method-gate" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614060 5577 cpu_manager.go:414] "containerMap: removing container" podUID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" containerName="perses" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614065 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" containerName="perses" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614077 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="prom-label-proxy" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614082 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="prom-label-proxy" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614099 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="kube-rbac-proxy" Oct 31 00:47:17.614102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="kube-rbac-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614110 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="prom-label-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614115 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="prom-label-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614120 5577 cpu_manager.go:414] "containerMap: removing container" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="config-reloader" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614124 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="config-reloader" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614130 5577 cpu_manager.go:414] "containerMap: removing container" podUID="336f26a3-2824-4671-8109-a023535f223c" containerName="otc-container" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614135 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="336f26a3-2824-4671-8109-a023535f223c" containerName="otc-container" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614144 5577 cpu_manager.go:414] "containerMap: removing container" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="init-config-reloader" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614148 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="init-config-reloader" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614158 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="method-gate" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614162 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="method-gate" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614169 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="kube-rbac-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614174 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="kube-rbac-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614180 5577 cpu_manager.go:414] "containerMap: removing container" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="alertmanager" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614184 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="alertmanager" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614235 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="kube-rbac-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614243 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="prom-label-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614248 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="config-reloader" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614253 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="336f26a3-2824-4671-8109-a023535f223c" containerName="otc-container" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614259 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="method-gate" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614265 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="kube-rbac-proxy" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614269 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf5e76cd-d3d3-415a-b48a-e53c33820757" containerName="alertmanager" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614275 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd87a0f-c9ac-409d-ace0-6900c286d135" containerName="method-gate" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614280 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b9c3c1-1d7c-4803-8c21-d5162648bc9a" containerName="perses" Oct 31 00:47:17.614314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.614285 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6dba107-55d3-4bbb-931c-36287b3603a9" containerName="prom-label-proxy" Oct 31 00:47:17.617024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.617002 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.621394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.621332 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 31 00:47:17.622294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622270 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:47:17.622294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622271 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:47:17.622500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622314 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-dockercfg-tjw7q\"" Oct 31 00:47:17.622500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622314 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:47:17.622500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622314 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-kube-rbac-proxy\"" Oct 31 00:47:17.622500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.622360 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-cluster-proxy-tls\"" Oct 31 00:47:17.626875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.626848 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:47:17.638297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.638272 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:47:17.641793 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.641770 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.655134 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.655110 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:47:17.662574 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.662548 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:47:17.665860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.665839 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.677648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.677627 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:47:17.680045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.680017 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:47:17.680156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.680055 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:47:17.680156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.680068 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-prwzv\"" Oct 31 00:47:17.680241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.680163 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-ff4x6\"" Oct 31 00:47:17.690421 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690398 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.690548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690425 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.690548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690447 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.690548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690464 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fllqw\" (UniqueName: \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.690668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690579 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.690668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690598 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.690668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690620 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.690865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690696 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.690865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690732 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbz98\" (UniqueName: \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.690865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.690825 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.691601 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.691575 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.722198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.722167 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:47:17.722332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.722212 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:47:17.722332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.722219 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:47:17.791234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791201 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791249 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791270 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbz98\" (UniqueName: \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791291 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791315 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791332 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791348 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791371 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfphq\" (UniqueName: \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.791418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791400 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.791894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791426 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fllqw\" (UniqueName: \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.791894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791468 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.791894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791501 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.791894 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.791529 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.792680 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:17.792296 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:47:17.792680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.792366 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.792680 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:17.792373 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert podName:936f8154-ccad-4d7a-b97e-719552066c92 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:18.292355271 +0000 UTC m=+2877.284014850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" (UID: "936f8154-ccad-4d7a-b97e-719552066c92") : secret "prometheus-operated-tls" not found Oct 31 00:47:17.792680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.792636 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.792680 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.792678 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.793065 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.792702 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.794304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.794248 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.794304 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.794282 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.795015 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.794995 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.804031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.804009 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbz98\" (UniqueName: \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-vgkqx\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.806867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.806848 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fllqw\" (UniqueName: \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:17.834909 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.834876 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.892025 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.891988 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.892925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.892130 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.892925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.892177 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfphq\" (UniqueName: \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.893443 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.892940 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.895983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.895949 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.905327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.905299 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfphq\" (UniqueName: \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\") pod \"data-science-thanos-querier-proxy-54fbc887bb-bf8rv\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:17.959379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.959337 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:47:17.966847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.966813 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:17.984788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:17.984754 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:47:18.094076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.094042 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.159939 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.159900 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:18.164610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.164588 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.173060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.172984 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:47:18.173887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.173863 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:18.176948 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.176927 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:18.177108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.177040 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.187483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.187439 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:18.194279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194257 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.194403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194286 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.194403 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194317 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc5d5\" (UniqueName: \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.194515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194488 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.194599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194584 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.194695 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.194682 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.237141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.237102 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.244211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.244188 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:47:18.246886 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:18.246859 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2771388_c994_4f83_b110_4d8fb5a4732e.slice/crio-cbff5256b73bd38553a76e6568c17bce2949e79dcc49744620d65744aa7ed1b0 WatchSource:0}: Error finding container cbff5256b73bd38553a76e6568c17bce2949e79dcc49744620d65744aa7ed1b0: Status 404 returned error can't find the container with id cbff5256b73bd38553a76e6568c17bce2949e79dcc49744620d65744aa7ed1b0 Oct 31 00:47:18.250279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250255 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:47:18.250480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250278 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9ht4b\"" Oct 31 00:47:18.250480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250326 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:47:18.250480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250371 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:47:18.250480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250400 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:47:18.250480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250448 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:47:18.250701 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.250545 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:47:18.272749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.272548 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:47:18.278774 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:18.278744 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffb6f4a7_4fa3_4538_92e4_78c644118a62.slice/crio-5494ae85bdd0713b417562fbf8109f6b20bd07e40816eb27db81bd7df9b8dbb7 WatchSource:0}: Error finding container 5494ae85bdd0713b417562fbf8109f6b20bd07e40816eb27db81bd7df9b8dbb7: Status 404 returned error can't find the container with id 5494ae85bdd0713b417562fbf8109f6b20bd07e40816eb27db81bd7df9b8dbb7 Oct 31 00:47:18.295604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295550 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.295604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295584 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.295821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295668 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.295821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295743 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:18.295821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295786 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr7cs\" (UniqueName: \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.296002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295821 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.296002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.295848 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.296002 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:18.295912 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/prometheus-operated-tls: secret "prometheus-operated-tls" not found Oct 31 00:47:18.296002 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:18.295936 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:47:18.296002 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:18.295987 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert podName:936f8154-ccad-4d7a-b97e-719552066c92 nodeName:}" failed. No retries permitted until 2026-10-31 00:47:19.29595906 +0000 UTC m=+2878.287618639 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "prometheus-client-cert" (UniqueName: "kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert") pod "data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" (UID: "936f8154-ccad-4d7a-b97e-719552066c92") : secret "prometheus-operated-tls" not found Oct 31 00:47:18.296273 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:18.296008 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs podName:2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf nodeName:}" failed. No retries permitted until 2026-10-31 00:47:18.795999142 +0000 UTC m=+2877.787658703 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:47:18.296317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.296302 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.296352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.296340 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.296414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.296391 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc5d5\" (UniqueName: \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.296808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.296759 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.298259 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.298239 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.298726 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.298626 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.298915 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.298899 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.313482 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.313458 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc5d5\" (UniqueName: \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.370976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.370937 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.397763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.397737 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr7cs\" (UniqueName: \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.397872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.397790 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.397872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.397858 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.398230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.398207 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.398546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.398501 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.411124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.411096 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr7cs\" (UniqueName: \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:18.641077 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.640982 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.801306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.801251 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.803970 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.803944 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:18.878924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.878889 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:18.907059 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.906946 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerStarted","Data":"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1"} Oct 31 00:47:18.907240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.907061 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerStarted","Data":"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2"} Oct 31 00:47:18.907240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.907098 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerStarted","Data":"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d"} Oct 31 00:47:18.907240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.907124 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerStarted","Data":"5494ae85bdd0713b417562fbf8109f6b20bd07e40816eb27db81bd7df9b8dbb7"} Oct 31 00:47:18.909342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.909313 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerStarted","Data":"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba"} Oct 31 00:47:18.909484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.909365 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerStarted","Data":"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d"} Oct 31 00:47:18.909484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.909388 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerStarted","Data":"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e"} Oct 31 00:47:18.909484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.909415 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerStarted","Data":"cbff5256b73bd38553a76e6568c17bce2949e79dcc49744620d65744aa7ed1b0"} Oct 31 00:47:18.928272 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:18.928212 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" podStartSLOduration=1.928200804 podStartE2EDuration="1.928200804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:18.927504052 +0000 UTC m=+2877.919163636" watchObservedRunningTime="2026-10-31 00:47:18.928200804 +0000 UTC m=+2877.919860388" Oct 31 00:47:19.038263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.038218 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.080510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.080485 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:19.277034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.276959 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:19.306925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.306893 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:19.309609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.309584 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"data-science-prometheus-cluster-proxy-545f759f7c-vwmkd\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:19.369885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.368908 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" podStartSLOduration=2.368884539 podStartE2EDuration="2.368884539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:18.951827931 +0000 UTC m=+2877.943487521" watchObservedRunningTime="2026-10-31 00:47:19.368884539 +0000 UTC m=+2878.360544125" Oct 31 00:47:19.369885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.369822 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:19.373580 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:19.373552 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1b0c23_a8a1_4adc_9f05_6d9e35295bdf.slice/crio-98e860e871f9689de9f2d308a6b35010b4360ce4adcaeb3727d93cf50d107bbd WatchSource:0}: Error finding container 98e860e871f9689de9f2d308a6b35010b4360ce4adcaeb3727d93cf50d107bbd: Status 404 returned error can't find the container with id 98e860e871f9689de9f2d308a6b35010b4360ce4adcaeb3727d93cf50d107bbd Oct 31 00:47:19.376091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.376056 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:19.376203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.376129 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:19.431648 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.431613 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:19.716828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.716638 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:47:19.719263 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:19.719235 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936f8154_ccad_4d7a_b97e_719552066c92.slice/crio-ee7d3e47b71f05ea1177f0c5ff31467bbdf9041fd5e9ad482b14aed354a6cc51 WatchSource:0}: Error finding container ee7d3e47b71f05ea1177f0c5ff31467bbdf9041fd5e9ad482b14aed354a6cc51: Status 404 returned error can't find the container with id ee7d3e47b71f05ea1177f0c5ff31467bbdf9041fd5e9ad482b14aed354a6cc51 Oct 31 00:47:19.915778 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.915743 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" event={"ID":"936f8154-ccad-4d7a-b97e-719552066c92","Type":"ContainerStarted","Data":"ee7d3e47b71f05ea1177f0c5ff31467bbdf9041fd5e9ad482b14aed354a6cc51"} Oct 31 00:47:19.917431 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.917403 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf","Type":"ContainerStarted","Data":"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340"} Oct 31 00:47:19.917554 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.917446 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf","Type":"ContainerStarted","Data":"98e860e871f9689de9f2d308a6b35010b4360ce4adcaeb3727d93cf50d107bbd"} Oct 31 00:47:19.939457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:19.939360 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.939346407 podStartE2EDuration="1.939346407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:19.938577702 +0000 UTC m=+2878.930237299" watchObservedRunningTime="2026-10-31 00:47:19.939346407 +0000 UTC m=+2878.931005991" Oct 31 00:47:20.919814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:20.919783 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:20.923523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:20.923478 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" event={"ID":"936f8154-ccad-4d7a-b97e-719552066c92","Type":"ContainerStarted","Data":"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7"} Oct 31 00:47:20.923664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:20.923542 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:20.927206 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:20.927186 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:20.943855 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:20.943802 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" podStartSLOduration=3.943785777 podStartE2EDuration="3.943785777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:20.942799899 +0000 UTC m=+2879.934459499" watchObservedRunningTime="2026-10-31 00:47:20.943785777 +0000 UTC m=+2879.935445362" Oct 31 00:47:21.021124 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.021085 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.021334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.021216 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.024757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.024729 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.117824 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.117788 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.144354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.144317 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.196020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.195938 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:47:21.203294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.203271 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.490341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.490268 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:47:21.498049 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:21.498015 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33ba49c3_a151_45b3_b6e1_fbf5ebcc246e.slice/crio-1cbb8b913b41ae01a0bef1000eee8538b12e294535952c8dbd8c84bf4c222445 WatchSource:0}: Error finding container 1cbb8b913b41ae01a0bef1000eee8538b12e294535952c8dbd8c84bf4c222445: Status 404 returned error can't find the container with id 1cbb8b913b41ae01a0bef1000eee8538b12e294535952c8dbd8c84bf4c222445 Oct 31 00:47:21.930314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.930274 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e","Type":"ContainerStarted","Data":"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad"} Oct 31 00:47:21.930827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.930329 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e","Type":"ContainerStarted","Data":"1cbb8b913b41ae01a0bef1000eee8538b12e294535952c8dbd8c84bf4c222445"} Oct 31 00:47:21.930827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.930609 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:21.950125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:21.950072 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=3.950057619 podStartE2EDuration="3.950057619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:21.949392177 +0000 UTC m=+2880.941051762" watchObservedRunningTime="2026-10-31 00:47:21.950057619 +0000 UTC m=+2880.941717202" Oct 31 00:47:24.479928 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.479890 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.697917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.697885 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.769650 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.769614 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:24.792868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.792833 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:24.793031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.792984 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.797326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.797300 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:47:24.797478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.797352 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:47:24.797478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.797355 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:47:24.797478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.797468 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rkqv7\"" Oct 31 00:47:24.797634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.797551 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:47:24.829555 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.829524 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:24.855004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.854975 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855009 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855102 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855139 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855158 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855225 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.855297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.855249 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnp9h\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.955936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.955897 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.955936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.955933 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956020 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956090 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956106 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956137 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956323 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnp9h\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.956557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.956532 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.959176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.959153 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.959338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.959155 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.959400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.959385 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.959689 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.959666 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.960141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.960118 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:24.969812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:24.969778 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnp9h\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:25.110641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:25.110531 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:25.228960 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:25.228916 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:25.406946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:25.406920 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:47:25.409576 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:25.409543 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded5d017f_86e9_4583_bac7_21290b0fc8e9.slice/crio-6b314f73caecd28c237bc603a51bf14508a82a326b50c51b7f68334e78793a04 WatchSource:0}: Error finding container 6b314f73caecd28c237bc603a51bf14508a82a326b50c51b7f68334e78793a04: Status 404 returned error can't find the container with id 6b314f73caecd28c237bc603a51bf14508a82a326b50c51b7f68334e78793a04 Oct 31 00:47:25.950293 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:25.950254 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerStarted","Data":"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7"} Oct 31 00:47:25.950732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:25.950318 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerStarted","Data":"6b314f73caecd28c237bc603a51bf14508a82a326b50c51b7f68334e78793a04"} Oct 31 00:47:26.932919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:26.932879 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:47:27.076698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:27.076665 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:29.292817 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:29.292728 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:47:29.969101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:29.969062 5577 generic.go:356] "Generic (PLEG): container finished" podID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerID="8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7" exitCode=0 Oct 31 00:47:29.969101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:29.969091 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerDied","Data":"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7"} Oct 31 00:47:30.976621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:30.976578 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerStarted","Data":"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f"} Oct 31 00:47:30.977068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:30.976659 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerStarted","Data":"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8"} Oct 31 00:47:30.977068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:30.976920 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:31.006519 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:31.006454 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.420710002 podStartE2EDuration="7.006436214s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.585726212s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:31.00430134 +0000 UTC m=+2889.995960923" watchObservedRunningTime="2026-10-31 00:47:31.006436214 +0000 UTC m=+2889.998095801" Oct 31 00:47:32.938109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:32.938081 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:47:36.982983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:36.982952 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:47:46.855257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:46.855188 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:46.988226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:46.988188 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.377932 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.377897 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:47.378109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.378092 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" containerName="otc-container" containerID="cri-o://d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340" gracePeriod=30 Oct 31 00:47:47.499314 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.499282 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.618693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.618668 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:47.627563 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.627535 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:47:47.764569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764541 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\" (UniqueName: \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.764791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764589 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.764791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764614 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.764791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764667 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.764791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764688 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.764791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.764782 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") pod \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\" (UID: \"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf\") " Oct 31 00:47:47.765196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.765136 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:47:47.767321 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.767295 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "kube-api-access-lc5d5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:47:47.767635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.767611 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:47.767739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.767636 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:47.767739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.767675 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:47.767858 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.767745 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs" pod "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" (UID: "2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:47:47.866145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866104 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:47.866145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866143 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:47.866573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866159 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:47.866573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866184 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc5d5\" (UniqueName: \"kubernetes.io/projected/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-kube-api-access-lc5d5\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:47.866573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866200 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:47.866573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:47.866215 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:47:48.048993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.048905 5577 generic.go:356] "Generic (PLEG): container finished" podID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" containerID="d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340" exitCode=0 Oct 31 00:47:48.048993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.048951 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.048993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.048970 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf","Type":"ContainerDied","Data":"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340"} Oct 31 00:47:48.049256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.049019 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf","Type":"ContainerDied","Data":"98e860e871f9689de9f2d308a6b35010b4360ce4adcaeb3727d93cf50d107bbd"} Oct 31 00:47:48.049256 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.049045 5577 scope.go:122] "RemoveContainer" containerID="d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340" Oct 31 00:47:48.060426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.060403 5577 scope.go:122] "RemoveContainer" containerID="d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340" Oct 31 00:47:48.060754 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:48.060734 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340\": container with ID starting with d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340 not found: ID does not exist" containerID="d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340" Oct 31 00:47:48.060813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.060767 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340"} err="failed to get container status \"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340\": rpc error: code = NotFound desc = could not find container \"d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340\": container with ID starting with d6ac234825dadd5b9dc24c15a6d45874b7412c1fa60d95c4429b4c6655fef340 not found: ID does not exist" Oct 31 00:47:48.072539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.072518 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:48.079476 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.079456 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:48.111604 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.111579 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:48.112004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.111990 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" containerName="otc-container" Oct 31 00:47:48.112093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.112006 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" containerName="otc-container" Oct 31 00:47:48.112093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.112075 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" containerName="otc-container" Oct 31 00:47:48.117199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.117165 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.121627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.121568 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:47:48.121812 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.121795 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:47:48.122165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.122142 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9ht4b\"" Oct 31 00:47:48.122241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.122151 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:47:48.122241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.122166 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:47:48.122241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.122151 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:47:48.126466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.125994 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:48.268722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.268673 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.268908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.268735 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbqm7\" (UniqueName: \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.268908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.268840 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.268908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.268869 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.268908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.268904 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.269078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.269061 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370108 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370150 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbqm7\" (UniqueName: \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370187 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370206 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370241 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.370492 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.370294 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.371382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.371346 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.373303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.373278 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.373493 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.373466 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.373622 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.373567 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.373692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.373667 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.382633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.382607 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbqm7\" (UniqueName: \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\") pod \"data-science-collector-collector-1\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.433457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.433433 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:47:48.715552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.715527 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:47:48.718009 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:47:48.717987 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf64101_231b_4677_8467_64c456aa2835.slice/crio-c288717858dd05429bb68bcfc751ed5338d1b9dd1e4d172a8614594577a609b2 WatchSource:0}: Error finding container c288717858dd05429bb68bcfc751ed5338d1b9dd1e4d172a8614594577a609b2: Status 404 returned error can't find the container with id c288717858dd05429bb68bcfc751ed5338d1b9dd1e4d172a8614594577a609b2 Oct 31 00:47:48.720329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.720296 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:48.720426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:48.720355 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:47:49.055487 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:49.055450 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abf64101-231b-4677-8467-64c456aa2835","Type":"ContainerStarted","Data":"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9"} Oct 31 00:47:49.056028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:49.055511 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abf64101-231b-4677-8467-64c456aa2835","Type":"ContainerStarted","Data":"c288717858dd05429bb68bcfc751ed5338d1b9dd1e4d172a8614594577a609b2"} Oct 31 00:47:49.076885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:49.076831 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.076818266 podStartE2EDuration="1.076818266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:49.076313452 +0000 UTC m=+2908.067973037" watchObservedRunningTime="2026-10-31 00:47:49.076818266 +0000 UTC m=+2908.068477849" Oct 31 00:47:49.133444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:47:49.133406 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf" path="/var/lib/kubelet/pods/2d1b0c23-a8a1-4adc-9f05-6d9e35295bdf/volumes" Oct 31 00:47:59.292421 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:47:59.292385 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:48:04.041501 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.041434 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.176212 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.176176 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.558200 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.558171 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:04.558453 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.558411 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="abf64101-231b-4677-8467-64c456aa2835" containerName="otc-container" containerID="cri-o://4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9" gracePeriod=30 Oct 31 00:48:04.678303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.678266 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:04.807884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:04.807843 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:05.810512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.810488 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:05.816618 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816600 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\" (UniqueName: \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.816731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816633 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.816731 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816659 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.816864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816764 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.816864 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816818 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\" (UniqueName: \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.816977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.816869 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\") pod \"abf64101-231b-4677-8467-64c456aa2835\" (UID: \"abf64101-231b-4677-8467-64c456aa2835\") " Oct 31 00:48:05.817240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.817212 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:05.819016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.818989 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "kube-api-access-fbqm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:05.819117 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.818989 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:05.819204 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.819185 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:05.819366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.819350 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:05.819429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.819371 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs" pod "abf64101-231b-4677-8467-64c456aa2835" (UID: "abf64101-231b-4677-8467-64c456aa2835"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917903 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917934 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbqm7\" (UniqueName: \"kubernetes.io/projected/abf64101-231b-4677-8467-64c456aa2835-kube-api-access-fbqm7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917946 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917954 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917964 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/abf64101-231b-4677-8467-64c456aa2835-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:05.917969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:05.917973 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/abf64101-231b-4677-8467-64c456aa2835-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:06.134811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.134775 5577 generic.go:356] "Generic (PLEG): container finished" podID="abf64101-231b-4677-8467-64c456aa2835" containerID="4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9" exitCode=0 Oct 31 00:48:06.134987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.134828 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.134987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.134874 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abf64101-231b-4677-8467-64c456aa2835","Type":"ContainerDied","Data":"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9"} Oct 31 00:48:06.134987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.134932 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"abf64101-231b-4677-8467-64c456aa2835","Type":"ContainerDied","Data":"c288717858dd05429bb68bcfc751ed5338d1b9dd1e4d172a8614594577a609b2"} Oct 31 00:48:06.134987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.134955 5577 scope.go:122] "RemoveContainer" containerID="4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9" Oct 31 00:48:06.144833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.144817 5577 scope.go:122] "RemoveContainer" containerID="4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9" Oct 31 00:48:06.145059 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:06.145040 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9\": container with ID starting with 4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9 not found: ID does not exist" containerID="4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9" Oct 31 00:48:06.145125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.145066 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9"} err="failed to get container status \"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9\": rpc error: code = NotFound desc = could not find container \"4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9\": container with ID starting with 4ee37ee04207f6877caeb841691ef6be3ad354f1b95a64bdc57a4a99bf372cb9 not found: ID does not exist" Oct 31 00:48:06.158150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.158128 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:06.166593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.166572 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:06.193884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.193834 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:06.194172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.194161 5577 cpu_manager.go:414] "containerMap: removing container" podUID="abf64101-231b-4677-8467-64c456aa2835" containerName="otc-container" Oct 31 00:48:06.194225 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.194174 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf64101-231b-4677-8467-64c456aa2835" containerName="otc-container" Oct 31 00:48:06.194263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.194237 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf64101-231b-4677-8467-64c456aa2835" containerName="otc-container" Oct 31 00:48:06.198484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.198469 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.202698 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.202677 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:06.203213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.203193 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:06.203297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.203193 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-9ht4b\"" Oct 31 00:48:06.203297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.203204 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:06.203297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.203253 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:06.203297 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.203255 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:06.208393 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.208372 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:06.221050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221028 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.221146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221059 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.221146 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221083 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.221231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221193 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.221317 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221300 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv27\" (UniqueName: \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.221377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.221360 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322257 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322296 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322302 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322329 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322379 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv27\" (UniqueName: \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322398 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.322578 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.322417 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.323128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.323096 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.325051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.325026 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.325157 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.325102 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.325538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.325518 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.325577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.325535 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.335089 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.335066 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv27\" (UniqueName: \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\") pod \"data-science-collector-collector-1\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.511441 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.511413 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:06.784793 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.784387 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:06.787326 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:06.787297 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b10ffd_0ab3_46a5_9327_c095f5e0393a.slice/crio-4cb801f3a9d330657572550eb9c140b43054b927bcc7c40956218448d419b239 WatchSource:0}: Error finding container 4cb801f3a9d330657572550eb9c140b43054b927bcc7c40956218448d419b239: Status 404 returned error can't find the container with id 4cb801f3a9d330657572550eb9c140b43054b927bcc7c40956218448d419b239 Oct 31 00:48:06.789674 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.789637 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:06.789804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:06.789741 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:07.129759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:07.129674 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf64101-231b-4677-8467-64c456aa2835" path="/var/lib/kubelet/pods/abf64101-231b-4677-8467-64c456aa2835/volumes" Oct 31 00:48:07.140557 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:07.140516 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d2b10ffd-0ab3-46a5-9327-c095f5e0393a","Type":"ContainerStarted","Data":"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527"} Oct 31 00:48:07.140728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:07.140569 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d2b10ffd-0ab3-46a5-9327-c095f5e0393a","Type":"ContainerStarted","Data":"4cb801f3a9d330657572550eb9c140b43054b927bcc7c40956218448d419b239"} Oct 31 00:48:07.162448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:07.162405 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.162392029 podStartE2EDuration="1.162392029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:07.161347079 +0000 UTC m=+2926.153006679" watchObservedRunningTime="2026-10-31 00:48:07.162392029 +0000 UTC m=+2926.154051612" Oct 31 00:48:10.287376 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.287342 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:48:10.287834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.287603 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" podUID="936f8154-ccad-4d7a-b97e-719552066c92" containerName="kube-rbac-proxy" containerID="cri-o://04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7" gracePeriod=30 Oct 31 00:48:10.290630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.290606 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:48:10.290838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.290819 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="kube-rbac-proxy" containerID="cri-o://cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" gracePeriod=30 Oct 31 00:48:10.290938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.290844 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="method-gate" containerID="cri-o://4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" gracePeriod=30 Oct 31 00:48:10.290938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.290846 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="prom-label-proxy" containerID="cri-o://1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" gracePeriod=30 Oct 31 00:48:10.294588 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.294563 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:48:10.294811 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.294792 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="kube-rbac-proxy" containerID="cri-o://16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" gracePeriod=30 Oct 31 00:48:10.294926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.294905 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="prom-label-proxy" containerID="cri-o://5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" gracePeriod=30 Oct 31 00:48:10.295009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.294993 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="method-gate" containerID="cri-o://82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" gracePeriod=30 Oct 31 00:48:10.507306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.497654 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.520922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.520894 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:10.521132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.521109 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" containerID="cri-o://3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8" gracePeriod=120 Oct 31 00:48:10.521411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.521375 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="config-reloader" containerID="cri-o://9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f" gracePeriod=120 Oct 31 00:48:10.632672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.632638 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.700217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.700123 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:48:10.701934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.701908 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:10.702116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.702090 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" containerName="otc-container" containerID="cri-o://fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527" gracePeriod=30 Oct 31 00:48:10.714211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.707423 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:48:10.714211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.711151 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:48:10.764732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.764623 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:10.767816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.767788 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:10.768000 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.767969 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" containerName="perses" containerID="cri-o://cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad" gracePeriod=30 Oct 31 00:48:10.776874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.776844 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\" (UniqueName: \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\") pod \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " Oct 31 00:48:10.777834 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.777814 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\") pod \"b2771388-c994-4f83-b110-4d8fb5a4732e\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " Oct 31 00:48:10.778067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.777856 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\") pod \"936f8154-ccad-4d7a-b97e-719552066c92\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " Oct 31 00:48:10.778067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.777885 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") pod \"936f8154-ccad-4d7a-b97e-719552066c92\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " Oct 31 00:48:10.778067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.777911 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\" (UniqueName: \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\") pod \"b2771388-c994-4f83-b110-4d8fb5a4732e\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " Oct 31 00:48:10.779002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.778589 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca" pod "b2771388-c994-4f83-b110-4d8fb5a4732e" (UID: "b2771388-c994-4f83-b110-4d8fb5a4732e"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.780699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780669 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\" (UniqueName: \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\") pod \"936f8154-ccad-4d7a-b97e-719552066c92\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " Oct 31 00:48:10.780842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780763 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\" (UniqueName: \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\") pod \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " Oct 31 00:48:10.780842 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780802 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\") pod \"936f8154-ccad-4d7a-b97e-719552066c92\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " Oct 31 00:48:10.780963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780877 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\") pod \"b2771388-c994-4f83-b110-4d8fb5a4732e\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " Oct 31 00:48:10.780963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780941 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\" (UniqueName: \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\") pod \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\" (UID: \"ffb6f4a7-4fa3-4538-92e4-78c644118a62\") " Oct 31 00:48:10.781068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.780979 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\" (UniqueName: \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\") pod \"b2771388-c994-4f83-b110-4d8fb5a4732e\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " Oct 31 00:48:10.781068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.781012 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\") pod \"b2771388-c994-4f83-b110-4d8fb5a4732e\" (UID: \"b2771388-c994-4f83-b110-4d8fb5a4732e\") " Oct 31 00:48:10.781189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.781110 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\") pod \"936f8154-ccad-4d7a-b97e-719552066c92\" (UID: \"936f8154-ccad-4d7a-b97e-719552066c92\") " Oct 31 00:48:10.781347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.781332 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.781510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.781485 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config" pod "b2771388-c994-4f83-b110-4d8fb5a4732e" (UID: "b2771388-c994-4f83-b110-4d8fb5a4732e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.782115 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls" pod "ffb6f4a7-4fa3-4538-92e4-78c644118a62" (UID: "ffb6f4a7-4fa3-4538-92e4-78c644118a62"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.782243 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config" pod "ffb6f4a7-4fa3-4538-92e4-78c644118a62" (UID: "ffb6f4a7-4fa3-4538-92e4-78c644118a62"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.782602 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca" pod "b2771388-c994-4f83-b110-4d8fb5a4732e" (UID: "b2771388-c994-4f83-b110-4d8fb5a4732e"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.783031 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca" pod "936f8154-ccad-4d7a-b97e-719552066c92" (UID: "936f8154-ccad-4d7a-b97e-719552066c92"). InnerVolumeSpecName "prometheus-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.783525 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls" pod "b2771388-c994-4f83-b110-4d8fb5a4732e" (UID: "b2771388-c994-4f83-b110-4d8fb5a4732e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.784113 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls" pod "936f8154-ccad-4d7a-b97e-719552066c92" (UID: "936f8154-ccad-4d7a-b97e-719552066c92"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.785210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.785092 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert" pod "936f8154-ccad-4d7a-b97e-719552066c92" (UID: "936f8154-ccad-4d7a-b97e-719552066c92"). InnerVolumeSpecName "prometheus-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.787553 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.787533 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw" pod "936f8154-ccad-4d7a-b97e-719552066c92" (UID: "936f8154-ccad-4d7a-b97e-719552066c92"). InnerVolumeSpecName "kube-api-access-fllqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.789582 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.789515 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq" pod "ffb6f4a7-4fa3-4538-92e4-78c644118a62" (UID: "ffb6f4a7-4fa3-4538-92e4-78c644118a62"). InnerVolumeSpecName "kube-api-access-jfphq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.790125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.789742 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config" pod "936f8154-ccad-4d7a-b97e-719552066c92" (UID: "936f8154-ccad-4d7a-b97e-719552066c92"). InnerVolumeSpecName "kube-rbac-proxy-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:10.791911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.791892 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98" pod "b2771388-c994-4f83-b110-4d8fb5a4732e" (UID: "b2771388-c994-4f83-b110-4d8fb5a4732e"). InnerVolumeSpecName "kube-api-access-zbz98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:10.882367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882338 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffb6f4a7-4fa3-4538-92e4-78c644118a62-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882369 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbz98\" (UniqueName: \"kubernetes.io/projected/b2771388-c994-4f83-b110-4d8fb5a4732e-kube-api-access-zbz98\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882385 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882399 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882411 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffb6f4a7-4fa3-4538-92e4-78c644118a62-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882423 5577 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-kube-rbac-proxy-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882435 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-client-cert\" (UniqueName: \"kubernetes.io/secret/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882447 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b2771388-c994-4f83-b110-4d8fb5a4732e-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882459 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fllqw\" (UniqueName: \"kubernetes.io/projected/936f8154-ccad-4d7a-b97e-719552066c92-kube-api-access-fllqw\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882472 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfphq\" (UniqueName: \"kubernetes.io/projected/ffb6f4a7-4fa3-4538-92e4-78c644118a62-kube-api-access-jfphq\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882497 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-ca\" (UniqueName: \"kubernetes.io/configmap/936f8154-ccad-4d7a-b97e-719552066c92-prometheus-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.882548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.882510 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b2771388-c994-4f83-b110-4d8fb5a4732e-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:10.916487 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:10.916430 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.061470 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.061437 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.068507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.068481 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:11.077884 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.077864 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:11.159162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.159131 5577 generic.go:356] "Generic (PLEG): container finished" podID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" containerID="fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527" exitCode=0 Oct 31 00:48:11.159324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.159180 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:11.159324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.159221 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d2b10ffd-0ab3-46a5-9327-c095f5e0393a","Type":"ContainerDied","Data":"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527"} Oct 31 00:48:11.159324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.159284 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d2b10ffd-0ab3-46a5-9327-c095f5e0393a","Type":"ContainerDied","Data":"4cb801f3a9d330657572550eb9c140b43054b927bcc7c40956218448d419b239"} Oct 31 00:48:11.159324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.159312 5577 scope.go:122] "RemoveContainer" containerID="fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527" Oct 31 00:48:11.161181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161158 5577 generic.go:356] "Generic (PLEG): container finished" podID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" exitCode=0 Oct 31 00:48:11.161181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161180 5577 generic.go:356] "Generic (PLEG): container finished" podID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" exitCode=0 Oct 31 00:48:11.161330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161190 5577 generic.go:356] "Generic (PLEG): container finished" podID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" exitCode=0 Oct 31 00:48:11.161330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161213 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" Oct 31 00:48:11.161330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161268 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerDied","Data":"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba"} Oct 31 00:48:11.161330 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161326 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerDied","Data":"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d"} Oct 31 00:48:11.161527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161366 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerDied","Data":"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e"} Oct 31 00:48:11.161527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.161398 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx" event={"ID":"b2771388-c994-4f83-b110-4d8fb5a4732e","Type":"ContainerDied","Data":"cbff5256b73bd38553a76e6568c17bce2949e79dcc49744620d65744aa7ed1b0"} Oct 31 00:48:11.164526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.164502 5577 generic.go:356] "Generic (PLEG): container finished" podID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerID="9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f" exitCode=0 Oct 31 00:48:11.164652 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.164573 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerDied","Data":"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f"} Oct 31 00:48:11.165784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.165764 5577 generic.go:356] "Generic (PLEG): container finished" podID="936f8154-ccad-4d7a-b97e-719552066c92" containerID="04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7" exitCode=0 Oct 31 00:48:11.165895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.165787 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" event={"ID":"936f8154-ccad-4d7a-b97e-719552066c92","Type":"ContainerDied","Data":"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7"} Oct 31 00:48:11.165895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.165807 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" Oct 31 00:48:11.165895 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.165824 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd" event={"ID":"936f8154-ccad-4d7a-b97e-719552066c92","Type":"ContainerDied","Data":"ee7d3e47b71f05ea1177f0c5ff31467bbdf9041fd5e9ad482b14aed354a6cc51"} Oct 31 00:48:11.168087 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168068 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" exitCode=0 Oct 31 00:48:11.168160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168089 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" exitCode=0 Oct 31 00:48:11.168160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168099 5577 generic.go:356] "Generic (PLEG): container finished" podID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" exitCode=0 Oct 31 00:48:11.168160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168111 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" Oct 31 00:48:11.168160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168147 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerDied","Data":"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1"} Oct 31 00:48:11.168353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168225 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerDied","Data":"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2"} Oct 31 00:48:11.168353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168276 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerDied","Data":"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d"} Oct 31 00:48:11.168353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.168310 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv" event={"ID":"ffb6f4a7-4fa3-4538-92e4-78c644118a62","Type":"ContainerDied","Data":"5494ae85bdd0713b417562fbf8109f6b20bd07e40816eb27db81bd7df9b8dbb7"} Oct 31 00:48:11.169460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169442 5577 scope.go:122] "RemoveContainer" containerID="fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527" Oct 31 00:48:11.169577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169511 5577 generic.go:356] "Generic (PLEG): container finished" podID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" containerID="cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad" exitCode=0 Oct 31 00:48:11.169577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169531 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e","Type":"ContainerDied","Data":"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad"} Oct 31 00:48:11.169577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169550 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:11.169577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169552 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e","Type":"ContainerDied","Data":"1cbb8b913b41ae01a0bef1000eee8538b12e294535952c8dbd8c84bf4c222445"} Oct 31 00:48:11.169887 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.169865 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527\": container with ID starting with fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527 not found: ID does not exist" containerID="fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527" Oct 31 00:48:11.169999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169899 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527"} err="failed to get container status \"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527\": rpc error: code = NotFound desc = could not find container \"fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527\": container with ID starting with fcfb94a1508090db4d6829ebf6663225a25b19b34427ec397ccf9a33d461e527 not found: ID does not exist" Oct 31 00:48:11.169999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.169921 5577 scope.go:122] "RemoveContainer" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" Oct 31 00:48:11.177795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.177779 5577 scope.go:122] "RemoveContainer" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" Oct 31 00:48:11.181382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.181364 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:48:11.185300 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185267 5577 scope.go:122] "RemoveContainer" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" Oct 31 00:48:11.185660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185642 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-vgkqx"] Oct 31 00:48:11.185787 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185688 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\" (UniqueName: \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.185919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185896 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.186013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185948 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.186013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.185975 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.186131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186012 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\" (UniqueName: \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\") pod \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " Oct 31 00:48:11.186131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186095 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " Oct 31 00:48:11.186244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186137 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\" (UniqueName: \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\") pod \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " Oct 31 00:48:11.186244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186167 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.186244 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186197 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\" (UniqueName: \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\") pod \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\" (UID: \"33ba49c3-a151-45b3-b6e1-fbf5ebcc246e\") " Oct 31 00:48:11.186471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186256 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\") pod \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\" (UID: \"d2b10ffd-0ab3-46a5-9327-c095f5e0393a\") " Oct 31 00:48:11.186989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.186891 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "kube-api-access-nqv27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.187120 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.187011 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.187120 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.187031 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config" pod "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" (UID: "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:11.188885 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.188858 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:11.189641 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.189616 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.189986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.189949 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.189986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.189967 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" (UID: "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:48:11.190531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.190504 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs" pod "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" (UID: "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e"). InnerVolumeSpecName "kube-api-access-fr7cs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:11.191196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.191172 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.192507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.192480 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs" pod "d2b10ffd-0ab3-46a5-9327-c095f5e0393a" (UID: "d2b10ffd-0ab3-46a5-9327-c095f5e0393a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:11.194764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.194746 5577 scope.go:122] "RemoveContainer" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" Oct 31 00:48:11.195060 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.195039 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": container with ID starting with 1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba not found: ID does not exist" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" Oct 31 00:48:11.195127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195067 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba"} err="failed to get container status \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": rpc error: code = NotFound desc = could not find container \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": container with ID starting with 1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba not found: ID does not exist" Oct 31 00:48:11.195127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195083 5577 scope.go:122] "RemoveContainer" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" Oct 31 00:48:11.195351 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.195328 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": container with ID starting with 4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d not found: ID does not exist" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" Oct 31 00:48:11.195398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195361 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d"} err="failed to get container status \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": rpc error: code = NotFound desc = could not find container \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": container with ID starting with 4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d not found: ID does not exist" Oct 31 00:48:11.195398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195383 5577 scope.go:122] "RemoveContainer" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" Oct 31 00:48:11.195658 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.195639 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": container with ID starting with cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e not found: ID does not exist" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" Oct 31 00:48:11.195825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195663 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e"} err="failed to get container status \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": rpc error: code = NotFound desc = could not find container \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": container with ID starting with cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e not found: ID does not exist" Oct 31 00:48:11.195825 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.195677 5577 scope.go:122] "RemoveContainer" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" Oct 31 00:48:11.196380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.196352 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba"} err="failed to get container status \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": rpc error: code = NotFound desc = could not find container \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": container with ID starting with 1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba not found: ID does not exist" Oct 31 00:48:11.196380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.196380 5577 scope.go:122] "RemoveContainer" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" Oct 31 00:48:11.201790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.199683 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d"} err="failed to get container status \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": rpc error: code = NotFound desc = could not find container \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": container with ID starting with 4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d not found: ID does not exist" Oct 31 00:48:11.201790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.199724 5577 scope.go:122] "RemoveContainer" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" Oct 31 00:48:11.201790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.200112 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e"} err="failed to get container status \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": rpc error: code = NotFound desc = could not find container \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": container with ID starting with cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e not found: ID does not exist" Oct 31 00:48:11.201790 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.200139 5577 scope.go:122] "RemoveContainer" containerID="1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba" Oct 31 00:48:11.202251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.202225 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba"} err="failed to get container status \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": rpc error: code = NotFound desc = could not find container \"1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba\": container with ID starting with 1343ca86df758ce9ef7678023f5d5103f46436917bc62f7bd0b67c3809a76fba not found: ID does not exist" Oct 31 00:48:11.202331 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.202253 5577 scope.go:122] "RemoveContainer" containerID="4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d" Oct 31 00:48:11.203573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.203543 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d"} err="failed to get container status \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": rpc error: code = NotFound desc = could not find container \"4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d\": container with ID starting with 4f4b0364a6c68e09c0bb4cc41d2970f520b76e66bfa30de61807544d9aba5e4d not found: ID does not exist" Oct 31 00:48:11.203573 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.203572 5577 scope.go:122] "RemoveContainer" containerID="cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e" Oct 31 00:48:11.204024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.204004 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e"} err="failed to get container status \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": rpc error: code = NotFound desc = could not find container \"cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e\": container with ID starting with cb3de9bedf25bc74b3c51e0dca48590cb9dc5dc179cf3d69b9c329e467c8759e not found: ID does not exist" Oct 31 00:48:11.204097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.204024 5577 scope.go:122] "RemoveContainer" containerID="04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7" Oct 31 00:48:11.208051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.208030 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:48:11.213438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.213417 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-cluster-proxy-545f759f7c-vwmkd"] Oct 31 00:48:11.213786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.213766 5577 scope.go:122] "RemoveContainer" containerID="04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7" Oct 31 00:48:11.214057 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.214040 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7\": container with ID starting with 04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7 not found: ID does not exist" containerID="04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7" Oct 31 00:48:11.214112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.214062 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7"} err="failed to get container status \"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7\": rpc error: code = NotFound desc = could not find container \"04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7\": container with ID starting with 04856a3adbad7db1a8ba552fb60febc828ab099bf8a39959d6a32d93ba37b5e7 not found: ID does not exist" Oct 31 00:48:11.214112 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.214076 5577 scope.go:122] "RemoveContainer" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" Oct 31 00:48:11.222574 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.222560 5577 scope.go:122] "RemoveContainer" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" Oct 31 00:48:11.226063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.226042 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:48:11.229673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.229659 5577 scope.go:122] "RemoveContainer" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" Oct 31 00:48:11.231253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.231236 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-bf8rv"] Oct 31 00:48:11.237075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237049 5577 scope.go:122] "RemoveContainer" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" Oct 31 00:48:11.237322 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.237304 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": container with ID starting with 5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1 not found: ID does not exist" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" Oct 31 00:48:11.237379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237330 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1"} err="failed to get container status \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": rpc error: code = NotFound desc = could not find container \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": container with ID starting with 5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1 not found: ID does not exist" Oct 31 00:48:11.237379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237346 5577 scope.go:122] "RemoveContainer" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" Oct 31 00:48:11.237578 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.237563 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": container with ID starting with 82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2 not found: ID does not exist" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" Oct 31 00:48:11.237620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237582 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2"} err="failed to get container status \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": rpc error: code = NotFound desc = could not find container \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": container with ID starting with 82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2 not found: ID does not exist" Oct 31 00:48:11.237620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237594 5577 scope.go:122] "RemoveContainer" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" Oct 31 00:48:11.237864 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.237847 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": container with ID starting with 16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d not found: ID does not exist" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" Oct 31 00:48:11.237911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237870 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d"} err="failed to get container status \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": rpc error: code = NotFound desc = could not find container \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": container with ID starting with 16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d not found: ID does not exist" Oct 31 00:48:11.237911 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.237883 5577 scope.go:122] "RemoveContainer" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" Oct 31 00:48:11.238094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238080 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1"} err="failed to get container status \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": rpc error: code = NotFound desc = could not find container \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": container with ID starting with 5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1 not found: ID does not exist" Oct 31 00:48:11.238144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238094 5577 scope.go:122] "RemoveContainer" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" Oct 31 00:48:11.238319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238302 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2"} err="failed to get container status \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": rpc error: code = NotFound desc = could not find container \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": container with ID starting with 82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2 not found: ID does not exist" Oct 31 00:48:11.238369 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238320 5577 scope.go:122] "RemoveContainer" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" Oct 31 00:48:11.238532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238516 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d"} err="failed to get container status \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": rpc error: code = NotFound desc = could not find container \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": container with ID starting with 16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d not found: ID does not exist" Oct 31 00:48:11.238586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238532 5577 scope.go:122] "RemoveContainer" containerID="5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1" Oct 31 00:48:11.238757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238740 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1"} err="failed to get container status \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": rpc error: code = NotFound desc = could not find container \"5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1\": container with ID starting with 5a73c687c6e6b9b4304abc0d738b564a0399913dbf2d05ff04aac961fddbbae1 not found: ID does not exist" Oct 31 00:48:11.238820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238757 5577 scope.go:122] "RemoveContainer" containerID="82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2" Oct 31 00:48:11.239009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.238988 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2"} err="failed to get container status \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": rpc error: code = NotFound desc = could not find container \"82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2\": container with ID starting with 82073c9a04375246503defded6454f7a883d79c81346551a491edee66c6eeba2 not found: ID does not exist" Oct 31 00:48:11.239009 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.239008 5577 scope.go:122] "RemoveContainer" containerID="16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d" Oct 31 00:48:11.239277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.239256 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d"} err="failed to get container status \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": rpc error: code = NotFound desc = could not find container \"16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d\": container with ID starting with 16cc25d6a35f27d6419da1b4aa4030ac0b6e731ded7c2b9462e65367a489c09d not found: ID does not exist" Oct 31 00:48:11.239277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.239274 5577 scope.go:122] "RemoveContainer" containerID="cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad" Oct 31 00:48:11.246228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.246208 5577 scope.go:122] "RemoveContainer" containerID="cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad" Oct 31 00:48:11.246463 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:11.246444 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad\": container with ID starting with cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad not found: ID does not exist" containerID="cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad" Oct 31 00:48:11.246529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.246471 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad"} err="failed to get container status \"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad\": rpc error: code = NotFound desc = could not find container \"cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad\": container with ID starting with cc21179ffa0f813c7d536e659ece0d6c01cda37206982942cac946f8788ce0ad not found: ID does not exist" Oct 31 00:48:11.287512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287490 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287514 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287524 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287551 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287561 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr7cs\" (UniqueName: \"kubernetes.io/projected/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-kube-api-access-fr7cs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287570 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287579 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287587 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.287868 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.287596 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqv27\" (UniqueName: \"kubernetes.io/projected/d2b10ffd-0ab3-46a5-9327-c095f5e0393a-kube-api-access-nqv27\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.300653 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.300632 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:48:11.343005 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.342977 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins" pod "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" (UID: "33ba49c3-a151-45b3-b6e1-fbf5ebcc246e"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:11.388612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.388586 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:11.388612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.388612 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:48:11.481749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.481724 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:11.486445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.486425 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:11.498353 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.498334 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:11.505259 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.505240 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:11.982681 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:11.982643 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.134:9093/-/ready\": dial tcp 10.133.0.134:9093: connect: connection refused" Oct 31 00:48:13.130995 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:13.130950 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" path="/var/lib/kubelet/pods/33ba49c3-a151-45b3-b6e1-fbf5ebcc246e/volumes" Oct 31 00:48:13.131585 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:13.131568 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="936f8154-ccad-4d7a-b97e-719552066c92" path="/var/lib/kubelet/pods/936f8154-ccad-4d7a-b97e-719552066c92/volumes" Oct 31 00:48:13.132145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:13.132128 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" path="/var/lib/kubelet/pods/b2771388-c994-4f83-b110-4d8fb5a4732e/volumes" Oct 31 00:48:13.133320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:13.133298 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" path="/var/lib/kubelet/pods/d2b10ffd-0ab3-46a5-9327-c095f5e0393a/volumes" Oct 31 00:48:13.133941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:13.133923 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" path="/var/lib/kubelet/pods/ffb6f4a7-4fa3-4538-92e4-78c644118a62/volumes" Oct 31 00:48:16.982808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:16.982767 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.134:9093/-/ready\": dial tcp 10.133.0.134:9093: connect: connection refused" Oct 31 00:48:18.295224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.295189 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.393565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.393535 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:48:18.393976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.393958 5577 cpu_manager.go:414] "containerMap: removing container" podUID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" containerName="perses" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.393977 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" containerName="perses" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394008 5577 cpu_manager.go:414] "containerMap: removing container" podUID="936f8154-ccad-4d7a-b97e-719552066c92" containerName="kube-rbac-proxy" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394016 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936f8154-ccad-4d7a-b97e-719552066c92" containerName="kube-rbac-proxy" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394034 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="method-gate" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394042 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="method-gate" Oct 31 00:48:18.394057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394054 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394061 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394072 5577 cpu_manager.go:414] "containerMap: removing container" podUID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" containerName="otc-container" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394079 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" containerName="otc-container" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394097 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394104 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394119 5577 cpu_manager.go:414] "containerMap: removing container" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394128 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394144 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394152 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394166 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="method-gate" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394173 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="method-gate" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394244 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="method-gate" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394258 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394268 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b10ffd-0ab3-46a5-9327-c095f5e0393a" containerName="otc-container" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394277 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="prom-label-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394287 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2771388-c994-4f83-b110-4d8fb5a4732e" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394296 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="kube-rbac-proxy" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394304 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffb6f4a7-4fa3-4538-92e4-78c644118a62" containerName="method-gate" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394315 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ba49c3-a151-45b3-b6e1-fbf5ebcc246e" containerName="perses" Oct 31 00:48:18.394374 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.394325 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="936f8154-ccad-4d7a-b97e-719552066c92" containerName="kube-rbac-proxy" Oct 31 00:48:18.397967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.397945 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.408139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.408118 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:48:18.412820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.412792 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:48:18.412938 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.412891 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.422576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.422547 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:48:18.449094 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.449065 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.462661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.462637 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-ms9cg\"" Oct 31 00:48:18.462661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.462649 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:48:18.462869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.462673 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-vn5bs\"" Oct 31 00:48:18.462869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.462795 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:48:18.486991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.486940 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:48:18.486991 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.486946 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:48:18.487183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.487054 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:48:18.487183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.487085 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:48:18.549221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549152 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.549221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549189 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.549350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549275 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnsbn\" (UniqueName: \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.549350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549322 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.549350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549340 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2vqm\" (UniqueName: \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.549580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549363 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.549580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549408 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.549580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.549439 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.590881 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.590836 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.650873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.650846 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.650877 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2vqm\" (UniqueName: \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.650902 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.650919 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.650980 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.651062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651052 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.651333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651073 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651117 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnsbn\" (UniqueName: \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.651742 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651698 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651754 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.651866 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651698 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.652015 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.651993 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.653451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.653432 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.653545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.653507 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.668231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.668203 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnsbn\" (UniqueName: \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\") pod \"data-science-thanos-querier-proxy-54fbc887bb-rrrlr\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.668435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.668417 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2vqm\" (UniqueName: \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-7rszl\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.716362 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.716339 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:48:18.721623 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.721591 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.730477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.730457 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:48:18.942358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.942261 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:18.947548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.947525 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:18.959438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.959408 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:18.960987 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.960952 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:18.965799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.964468 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:18.968076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.968055 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:18.971103 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.971055 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:18.979037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:18.978957 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:19.015190 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.015159 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:48:19.019107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.018389 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:48:19.022953 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:19.022927 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6113840e_b89c_4c86_b33b_3d4cc6c1be47.slice/crio-566e0756dbd9125f52b88a8dee1b0f5d91420d26891ac597dad3d8bf59d1908b WatchSource:0}: Error finding container 566e0756dbd9125f52b88a8dee1b0f5d91420d26891ac597dad3d8bf59d1908b: Status 404 returned error can't find the container with id 566e0756dbd9125f52b88a8dee1b0f5d91420d26891ac597dad3d8bf59d1908b Oct 31 00:48:19.025266 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:19.025239 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d7e9e4_eeb8_42ac_aee1_db3a04662469.slice/crio-f24df87493a035aaeb6e09741f644c18deb2585148c6c694587531c00abbcb6a WatchSource:0}: Error finding container f24df87493a035aaeb6e09741f644c18deb2585148c6c694587531c00abbcb6a: Status 404 returned error can't find the container with id f24df87493a035aaeb6e09741f644c18deb2585148c6c694587531c00abbcb6a Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038306 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038349 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038367 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zfncz\"" Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038395 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038402 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:48:19.038423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038394 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:19.038687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.038457 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:19.053997 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.053978 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.054075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054012 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.054075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054037 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.054160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054144 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.054201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054169 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs286\" (UniqueName: \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.054201 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054184 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.054262 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054220 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.054294 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054265 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km5lg\" (UniqueName: \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.054324 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.054305 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.105550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.105522 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.154935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154815 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.154935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154877 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.154935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154908 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs286\" (UniqueName: \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.154935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154932 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.155177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154967 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.155177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.154995 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km5lg\" (UniqueName: \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.155177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155038 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.155177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155063 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.155177 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155103 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.155428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155189 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.155428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:19.155257 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:19.155428 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:19.155312 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs podName:a7a74d79-8e7e-43a6-94f8-5469a76dda02 nodeName:}" failed. No retries permitted until 2026-10-31 00:48:19.655290238 +0000 UTC m=+2938.646949817 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:48:19.155638 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155616 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.155947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.155928 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.157485 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.157463 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.158093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.158071 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.158240 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.158132 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.170207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.170181 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs286\" (UniqueName: \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.170872 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.170849 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km5lg\" (UniqueName: \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:19.209830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.209802 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerStarted","Data":"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a"} Oct 31 00:48:19.209965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.209865 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerStarted","Data":"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d"} Oct 31 00:48:19.209965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.209898 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerStarted","Data":"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc"} Oct 31 00:48:19.209965 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.209932 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerStarted","Data":"f24df87493a035aaeb6e09741f644c18deb2585148c6c694587531c00abbcb6a"} Oct 31 00:48:19.211846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.211818 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerStarted","Data":"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9"} Oct 31 00:48:19.211949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.211894 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerStarted","Data":"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b"} Oct 31 00:48:19.211949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.211938 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerStarted","Data":"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d"} Oct 31 00:48:19.212051 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.211984 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerStarted","Data":"566e0756dbd9125f52b88a8dee1b0f5d91420d26891ac597dad3d8bf59d1908b"} Oct 31 00:48:19.230203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.230151 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" podStartSLOduration=1.230139712 podStartE2EDuration="1.230139712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:19.229799155 +0000 UTC m=+2938.221458764" watchObservedRunningTime="2026-10-31 00:48:19.230139712 +0000 UTC m=+2938.221799295" Oct 31 00:48:19.252875 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.252824 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" podStartSLOduration=1.252806047 podStartE2EDuration="1.252806047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:19.251799422 +0000 UTC m=+2938.243459007" watchObservedRunningTime="2026-10-31 00:48:19.252806047 +0000 UTC m=+2938.244465632" Oct 31 00:48:19.348577 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.348543 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.660562 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.660532 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.663219 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.663197 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.750050 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.750017 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:19.866565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.866536 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:19.956320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:19.956283 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:20.149791 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.149768 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:20.152512 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:20.152479 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7a74d79_8e7e_43a6_94f8_5469a76dda02.slice/crio-993a257ba522b4a7e7b354126ff4fc157f13a8bd19a8f87fc67493fd51fd0fc5 WatchSource:0}: Error finding container 993a257ba522b4a7e7b354126ff4fc157f13a8bd19a8f87fc67493fd51fd0fc5: Status 404 returned error can't find the container with id 993a257ba522b4a7e7b354126ff4fc157f13a8bd19a8f87fc67493fd51fd0fc5 Oct 31 00:48:20.155024 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.154973 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:20.155132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.155074 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:20.219414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.219382 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7a74d79-8e7e-43a6-94f8-5469a76dda02","Type":"ContainerStarted","Data":"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639"} Oct 31 00:48:20.219569 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.219444 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7a74d79-8e7e-43a6-94f8-5469a76dda02","Type":"ContainerStarted","Data":"993a257ba522b4a7e7b354126ff4fc157f13a8bd19a8f87fc67493fd51fd0fc5"} Oct 31 00:48:20.242226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.242177 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.242163123 podStartE2EDuration="2.242163123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:20.241132917 +0000 UTC m=+2939.232792494" watchObservedRunningTime="2026-10-31 00:48:20.242163123 +0000 UTC m=+2939.233822707" Oct 31 00:48:20.760899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.760878 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:20.872013 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.871938 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872012 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872040 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872072 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872092 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872230 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872271 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\") pod \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\" (UID: \"ed5d017f-86e9-4583-bac7-21290b0fc8e9\") " Oct 31 00:48:20.872518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872437 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:20.872612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.872592 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.874874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.874842 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:20.875001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.874930 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:20.875097 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.875073 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "kube-api-access-vnp9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:20.875194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.875171 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:20.878838 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.878816 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:20.886039 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.886016 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config" pod "ed5d017f-86e9-4583-bac7-21290b0fc8e9" (UID: "ed5d017f-86e9-4583-bac7-21290b0fc8e9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:20.973594 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973563 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.973594 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973589 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.973808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973601 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.973808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973615 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ed5d017f-86e9-4583-bac7-21290b0fc8e9-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.973808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973627 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:20.973808 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:20.973643 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnp9h\" (UniqueName: \"kubernetes.io/projected/ed5d017f-86e9-4583-bac7-21290b0fc8e9-kube-api-access-vnp9h\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:21.226082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.226046 5577 generic.go:356] "Generic (PLEG): container finished" podID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerID="3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8" exitCode=0 Oct 31 00:48:21.226238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.226115 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerDied","Data":"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8"} Oct 31 00:48:21.226238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.226123 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:21.226238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.226203 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"ed5d017f-86e9-4583-bac7-21290b0fc8e9","Type":"ContainerDied","Data":"6b314f73caecd28c237bc603a51bf14508a82a326b50c51b7f68334e78793a04"} Oct 31 00:48:21.226238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.226233 5577 scope.go:122] "RemoveContainer" containerID="9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f" Oct 31 00:48:21.234561 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.234541 5577 scope.go:122] "RemoveContainer" containerID="3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8" Oct 31 00:48:21.241899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.241877 5577 scope.go:122] "RemoveContainer" containerID="8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7" Oct 31 00:48:21.247933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.247910 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:21.249326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.249312 5577 scope.go:122] "RemoveContainer" containerID="9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f" Oct 31 00:48:21.249609 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:21.249576 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f\": container with ID starting with 9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f not found: ID does not exist" containerID="9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f" Oct 31 00:48:21.249692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.249619 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f"} err="failed to get container status \"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f\": rpc error: code = NotFound desc = could not find container \"9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f\": container with ID starting with 9e6edfe50aafa08811547cd6debc0d5a880acd9761f60b3cd18a12a2ca3a6b7f not found: ID does not exist" Oct 31 00:48:21.249692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.249643 5577 scope.go:122] "RemoveContainer" containerID="3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8" Oct 31 00:48:21.250208 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:21.250176 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8\": container with ID starting with 3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8 not found: ID does not exist" containerID="3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8" Oct 31 00:48:21.250280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.250212 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8"} err="failed to get container status \"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8\": rpc error: code = NotFound desc = could not find container \"3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8\": container with ID starting with 3db0581cef4a01583b340ff4723d1b1ce706ba0b0a1b201a1fa62ce5b44ec4c8 not found: ID does not exist" Oct 31 00:48:21.250280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.250227 5577 scope.go:122] "RemoveContainer" containerID="8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7" Oct 31 00:48:21.250481 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:21.250463 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7\": container with ID starting with 8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7 not found: ID does not exist" containerID="8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7" Oct 31 00:48:21.250526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.250486 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7"} err="failed to get container status \"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7\": rpc error: code = NotFound desc = could not find container \"8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7\": container with ID starting with 8bd58e91f1b66bdf29b9d991cc25843232ebaca8f1609adf937dd692471b90c7 not found: ID does not exist" Oct 31 00:48:21.252986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:21.252968 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:22.790728 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.790674 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:22.798061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.798031 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:22.891777 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.891739 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:22.891933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.891871 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:22.895581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.895563 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:22.987430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:22.987387 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:23.006428 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:23.006405 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:23.130027 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:23.129955 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" path="/var/lib/kubelet/pods/ed5d017f-86e9-4583-bac7-21290b0fc8e9/volumes" Oct 31 00:48:23.188133 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:23.188109 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:48:23.196350 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:23.196319 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:23.473332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:23.473244 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:48:23.476994 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:23.476963 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2572119d_0db7_4d48_8224_e32d9deba663.slice/crio-68ed043c5cfe5030ee9ab981050e00dd616a944a3dce8ed97c8468c44147a303 WatchSource:0}: Error finding container 68ed043c5cfe5030ee9ab981050e00dd616a944a3dce8ed97c8468c44147a303: Status 404 returned error can't find the container with id 68ed043c5cfe5030ee9ab981050e00dd616a944a3dce8ed97c8468c44147a303 Oct 31 00:48:24.240148 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.240112 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"2572119d-0db7-4d48-8224-e32d9deba663","Type":"ContainerStarted","Data":"18c0567b9d9a863128c515f49ed02b3c4be0343c91b81d84aebba21d7011f700"} Oct 31 00:48:24.240550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.240164 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"2572119d-0db7-4d48-8224-e32d9deba663","Type":"ContainerStarted","Data":"68ed043c5cfe5030ee9ab981050e00dd616a944a3dce8ed97c8468c44147a303"} Oct 31 00:48:24.240550 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.240185 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:24.259490 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.259433 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.259417255 podStartE2EDuration="6.259417255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:24.258588578 +0000 UTC m=+2943.250248164" watchObservedRunningTime="2026-10-31 00:48:24.259417255 +0000 UTC m=+2943.251076838" Oct 31 00:48:24.597946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.597911 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:24.809800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.809753 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:24.947982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:24.947393 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:25.078635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.078600 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:25.123211 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123170 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:25.123736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123693 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" Oct 31 00:48:25.123736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123729 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" Oct 31 00:48:25.123904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123783 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="init-config-reloader" Oct 31 00:48:25.123904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123793 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="init-config-reloader" Oct 31 00:48:25.123904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123809 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="config-reloader" Oct 31 00:48:25.123904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123818 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="config-reloader" Oct 31 00:48:25.123904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123898 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="config-reloader" Oct 31 00:48:25.124153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.123912 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed5d017f-86e9-4583-bac7-21290b0fc8e9" containerName="alertmanager" Oct 31 00:48:25.144108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.144080 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.149523 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.149491 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:48:25.149814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.149790 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:48:25.150821 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.149957 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:48:25.151478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.151338 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:48:25.152935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.152784 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-sc6dq\"" Oct 31 00:48:25.167720 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.165792 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:25.212614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.212535 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.212614 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.212587 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqjgw\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.212897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.212640 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.212897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.212690 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.212897 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.212743 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.213073 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.213034 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.213127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.213098 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.314589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314556 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314608 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqjgw\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314639 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314669 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314699 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314854 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.314894 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.315565 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.315454 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.318040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.318007 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.318955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.318924 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.319058 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.318957 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.319968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.319878 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.320214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.320150 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.330761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.328219 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqjgw\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.396590 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.396556 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:25.478414 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.478329 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:25.771433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:25.771342 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:48:25.774911 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:25.774886 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d2ae77e_61af_4549_962d_14f168afaf50.slice/crio-2c67dd41ee71112c0f00ad31d06fb8d19578be23c49f087f750cc704ee7514dd WatchSource:0}: Error finding container 2c67dd41ee71112c0f00ad31d06fb8d19578be23c49f087f750cc704ee7514dd: Status 404 returned error can't find the container with id 2c67dd41ee71112c0f00ad31d06fb8d19578be23c49f087f750cc704ee7514dd Oct 31 00:48:26.249937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:26.249905 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerStarted","Data":"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6"} Oct 31 00:48:26.250126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:26.249960 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerStarted","Data":"2c67dd41ee71112c0f00ad31d06fb8d19578be23c49f087f750cc704ee7514dd"} Oct 31 00:48:27.146908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:27.146866 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:29.292117 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:29.292083 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:48:30.267439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:30.267405 5577 generic.go:356] "Generic (PLEG): container finished" podID="2d2ae77e-61af-4549-962d-14f168afaf50" containerID="61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6" exitCode=0 Oct 31 00:48:30.267612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:30.267477 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerDied","Data":"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6"} Oct 31 00:48:31.277832 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:31.277799 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerStarted","Data":"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826"} Oct 31 00:48:31.278265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:31.277857 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerStarted","Data":"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec"} Oct 31 00:48:31.278265 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:31.278042 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:31.307830 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:31.306505 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.82112552 podStartE2EDuration="6.306488007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.485362487s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:31.304790758 +0000 UTC m=+2950.296450343" watchObservedRunningTime="2026-10-31 00:48:31.306488007 +0000 UTC m=+2950.298147592" Oct 31 00:48:35.247070 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:35.247043 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:48:37.283529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:37.283501 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:48:46.358848 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:46.358803 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.484575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:46.484538 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:46.845899 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:46.845861 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:46.846223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:46.846188 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" containerName="otc-container" containerID="cri-o://d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639" gracePeriod=30 Oct 31 00:48:46.961747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:46.961685 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.090411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.090385 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.096730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.096642 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:47.194319 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194288 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194328 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194353 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\" (UniqueName: \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194379 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194448 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194469 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\") pod \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\" (UID: \"a7a74d79-8e7e-43a6-94f8-5469a76dda02\") " Oct 31 00:48:47.194818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.194773 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:47.197060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.197024 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "kube-api-access-zs286". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:47.197060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.197034 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:47.197245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.197057 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:47.197301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.197263 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:47.197536 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.197513 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs" pod "a7a74d79-8e7e-43a6-94f8-5469a76dda02" (UID: "a7a74d79-8e7e-43a6-94f8-5469a76dda02"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:47.295609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295580 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.295609 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295605 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.295822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295617 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.295822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295626 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.295822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295634 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/a7a74d79-8e7e-43a6-94f8-5469a76dda02-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.295822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.295644 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs286\" (UniqueName: \"kubernetes.io/projected/a7a74d79-8e7e-43a6-94f8-5469a76dda02-kube-api-access-zs286\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:47.343664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.343630 5577 generic.go:356] "Generic (PLEG): container finished" podID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" containerID="d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639" exitCode=0 Oct 31 00:48:47.343835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.343687 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.343835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.343734 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7a74d79-8e7e-43a6-94f8-5469a76dda02","Type":"ContainerDied","Data":"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639"} Oct 31 00:48:47.343835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.343804 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"a7a74d79-8e7e-43a6-94f8-5469a76dda02","Type":"ContainerDied","Data":"993a257ba522b4a7e7b354126ff4fc157f13a8bd19a8f87fc67493fd51fd0fc5"} Oct 31 00:48:47.343835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.343828 5577 scope.go:122] "RemoveContainer" containerID="d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639" Oct 31 00:48:47.354358 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.354335 5577 scope.go:122] "RemoveContainer" containerID="d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639" Oct 31 00:48:47.354768 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:47.354736 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639\": container with ID starting with d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639 not found: ID does not exist" containerID="d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639" Oct 31 00:48:47.354840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.354778 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639"} err="failed to get container status \"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639\": rpc error: code = NotFound desc = could not find container \"d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639\": container with ID starting with d20792091749fa3c63412a60e4338bdd6ba2b08c4379fc1aeca7f06d9f3d4639 not found: ID does not exist" Oct 31 00:48:47.373411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.373386 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:47.382302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.382250 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:47.408844 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.408819 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:47.409183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.409170 5577 cpu_manager.go:414] "containerMap: removing container" podUID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" containerName="otc-container" Oct 31 00:48:47.409231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.409185 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" containerName="otc-container" Oct 31 00:48:47.409267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.409263 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" containerName="otc-container" Oct 31 00:48:47.413744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.413725 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.418002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.417980 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:48:47.418378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.418355 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:48:47.418504 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.418473 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zfncz\"" Oct 31 00:48:47.418581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.418562 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:48:47.418702 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.418658 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:48:47.418799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.418745 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:48:47.425746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.425691 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:47.497852 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.497824 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.497852 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.497855 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.498071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.497872 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.498071 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.497901 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm7n4\" (UniqueName: \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.498172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.498114 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.498231 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.498210 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599202 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599240 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599246 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599255 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599286 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm7n4\" (UniqueName: \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599409 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.599538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.599457 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.600183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.600124 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.602173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.602149 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.602301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.602183 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.602444 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.602429 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.602481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.602448 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.612135 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.612084 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm7n4\" (UniqueName: \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\") pod \"data-science-collector-collector-1\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:47.729651 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:47.729617 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:48:48.007671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.007588 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:48.010790 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:48:48.010758 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6486a0f1_20f2_445f_a736_f4311431e97e.slice/crio-90b26acbc43d33cf8b8aa45fe5e22298427d57f7bf2abe765d2b3d0d922074c0 WatchSource:0}: Error finding container 90b26acbc43d33cf8b8aa45fe5e22298427d57f7bf2abe765d2b3d0d922074c0: Status 404 returned error can't find the container with id 90b26acbc43d33cf8b8aa45fe5e22298427d57f7bf2abe765d2b3d0d922074c0 Oct 31 00:48:48.013108 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.013074 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:48.013228 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.013138 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:48:48.350929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.350831 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6486a0f1-20f2-445f-a736-f4311431e97e","Type":"ContainerStarted","Data":"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1"} Oct 31 00:48:48.350929 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.350898 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6486a0f1-20f2-445f-a736-f4311431e97e","Type":"ContainerStarted","Data":"90b26acbc43d33cf8b8aa45fe5e22298427d57f7bf2abe765d2b3d0d922074c0"} Oct 31 00:48:48.373160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:48.373109 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.373094522 podStartE2EDuration="1.373094522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:48.372082743 +0000 UTC m=+2967.363742327" watchObservedRunningTime="2026-10-31 00:48:48.373094522 +0000 UTC m=+2967.364754106" Oct 31 00:48:49.129382 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:49.129346 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7a74d79-8e7e-43a6-94f8-5469a76dda02" path="/var/lib/kubelet/pods/a7a74d79-8e7e-43a6-94f8-5469a76dda02/volumes" Oct 31 00:48:58.380328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:58.380291 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:58.578969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:58.578934 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:58.850656 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:58.850622 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:48:58.850889 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:58.850866 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="6486a0f1-20f2-445f-a736-f4311431e97e" containerName="otc-container" containerID="cri-o://a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1" gracePeriod=30 Oct 31 00:48:58.968109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:58.968057 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:59.106329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:48:59.106251 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:48:59.294926 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:48:59.294892 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:49:00.096869 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.096848 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.206488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206415 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.206488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206455 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\" (UniqueName: \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.206488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206475 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.206799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206510 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.206799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206529 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.206799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.206549 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\") pod \"6486a0f1-20f2-445f-a736-f4311431e97e\" (UID: \"6486a0f1-20f2-445f-a736-f4311431e97e\") " Oct 31 00:49:00.207235 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.207200 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:00.208947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.208919 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:00.209376 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.209358 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "kube-api-access-zm7n4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:00.209429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.209380 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:00.209491 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.209454 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:00.209538 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.209484 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs" pod "6486a0f1-20f2-445f-a736-f4311431e97e" (UID: "6486a0f1-20f2-445f-a736-f4311431e97e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:00.307980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.307955 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.307980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.307978 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.308172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.307994 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.308172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.308005 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/6486a0f1-20f2-445f-a736-f4311431e97e-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.308172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.308014 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/6486a0f1-20f2-445f-a736-f4311431e97e-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.308172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.308023 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm7n4\" (UniqueName: \"kubernetes.io/projected/6486a0f1-20f2-445f-a736-f4311431e97e-kube-api-access-zm7n4\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:00.412221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.412190 5577 generic.go:356] "Generic (PLEG): container finished" podID="6486a0f1-20f2-445f-a736-f4311431e97e" containerID="a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1" exitCode=0 Oct 31 00:49:00.412344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.412237 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.412344 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.412281 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6486a0f1-20f2-445f-a736-f4311431e97e","Type":"ContainerDied","Data":"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1"} Oct 31 00:49:00.412424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.412344 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"6486a0f1-20f2-445f-a736-f4311431e97e","Type":"ContainerDied","Data":"90b26acbc43d33cf8b8aa45fe5e22298427d57f7bf2abe765d2b3d0d922074c0"} Oct 31 00:49:00.412424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.412372 5577 scope.go:122] "RemoveContainer" containerID="a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1" Oct 31 00:49:00.422042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.422022 5577 scope.go:122] "RemoveContainer" containerID="a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1" Oct 31 00:49:00.422294 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:00.422277 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1\": container with ID starting with a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1 not found: ID does not exist" containerID="a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1" Oct 31 00:49:00.422356 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.422301 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1"} err="failed to get container status \"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1\": rpc error: code = NotFound desc = could not find container \"a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1\": container with ID starting with a616acf2beefe97902affb22dadef36907cd01a5a416953f776ffad1608b18c1 not found: ID does not exist" Oct 31 00:49:00.434355 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.434334 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:00.440853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.440835 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:00.466959 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.466904 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:00.467220 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.467209 5577 cpu_manager.go:414] "containerMap: removing container" podUID="6486a0f1-20f2-445f-a736-f4311431e97e" containerName="otc-container" Oct 31 00:49:00.467263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.467222 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6486a0f1-20f2-445f-a736-f4311431e97e" containerName="otc-container" Oct 31 00:49:00.467305 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.467296 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="6486a0f1-20f2-445f-a736-f4311431e97e" containerName="otc-container" Oct 31 00:49:00.470207 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.470193 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.474152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474131 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zfncz\"" Oct 31 00:49:00.474340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474264 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:00.474340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474309 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:00.474340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474313 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:00.474484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474363 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:00.474749 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.474735 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:00.480352 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.480330 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:00.611328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611296 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mdq9\" (UniqueName: \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.611481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611333 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.611481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611353 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.611481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611461 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.611622 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611487 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.611622 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.611517 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712647 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712698 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712770 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712813 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mdq9\" (UniqueName: \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712848 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.712880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.712875 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.713584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.713561 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.715367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.715346 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.715823 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.715798 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.715916 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.715863 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.716004 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.715927 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.725751 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.725663 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mdq9\" (UniqueName: \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\") pod \"data-science-collector-collector-1\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:00.783668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:00.783634 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:01.057841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.057811 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:01.060980 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:01.060954 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70db1a15_ce89_40a9_ae87_b1d2ce967f2d.slice/crio-cabbfd52b41178e5c6c9cfd50382c0bb11b237415c18070a13f72300f2452118 WatchSource:0}: Error finding container cabbfd52b41178e5c6c9cfd50382c0bb11b237415c18070a13f72300f2452118: Status 404 returned error can't find the container with id cabbfd52b41178e5c6c9cfd50382c0bb11b237415c18070a13f72300f2452118 Oct 31 00:49:01.063576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.063544 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:01.063675 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.063613 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:01.129797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.129772 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6486a0f1-20f2-445f-a736-f4311431e97e" path="/var/lib/kubelet/pods/6486a0f1-20f2-445f-a736-f4311431e97e/volumes" Oct 31 00:49:01.417673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.417583 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"70db1a15-ce89-40a9-ae87-b1d2ce967f2d","Type":"ContainerStarted","Data":"cbe1f8b3df4ab495cd5d192c0c9cc7062277de97ff08771e3b5e04ef4751b30b"} Oct 31 00:49:01.417673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.417642 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"70db1a15-ce89-40a9-ae87-b1d2ce967f2d","Type":"ContainerStarted","Data":"cabbfd52b41178e5c6c9cfd50382c0bb11b237415c18070a13f72300f2452118"} Oct 31 00:49:01.438972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:01.438915 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.438902661 podStartE2EDuration="1.438902661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:01.438072804 +0000 UTC m=+2980.429732401" watchObservedRunningTime="2026-10-31 00:49:01.438902661 +0000 UTC m=+2980.430562308" Oct 31 00:49:04.073920 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.073879 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:04.837598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.837334 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:49:04.837827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.837671 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="kube-rbac-proxy" containerID="cri-o://d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" gracePeriod=30 Oct 31 00:49:04.837827 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.837759 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="prom-label-proxy" containerID="cri-o://0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" gracePeriod=30 Oct 31 00:49:04.837962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.837832 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="method-gate" containerID="cri-o://20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" gracePeriod=30 Oct 31 00:49:04.838583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.838361 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:49:04.838630 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.838609 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="kube-rbac-proxy" containerID="cri-o://6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" gracePeriod=30 Oct 31 00:49:04.838764 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.838740 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="method-gate" containerID="cri-o://7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" gracePeriod=30 Oct 31 00:49:04.838917 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:04.838752 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="prom-label-proxy" containerID="cri-o://68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" gracePeriod=30 Oct 31 00:49:05.027962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.027877 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.049011 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.048974 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:05.049463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.049214 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" containerID="cri-o://1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec" gracePeriod=120 Oct 31 00:49:05.049463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.049339 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="config-reloader" containerID="cri-o://38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826" gracePeriod=120 Oct 31 00:49:05.157784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.157764 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:49:05.160548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.160530 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:49:05.168637 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.167884 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.237839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.237804 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:05.238040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.238015 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" containerName="otc-container" containerID="cri-o://cbe1f8b3df4ab495cd5d192c0c9cc7062277de97ff08771e3b5e04ef4751b30b" gracePeriod=30 Oct 31 00:49:05.258415 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.258381 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\" (UniqueName: \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\") pod \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " Oct 31 00:49:05.258571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.258452 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\" (UniqueName: \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\") pod \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " Oct 31 00:49:05.258571 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.258553 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\" (UniqueName: \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\") pod \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\" (UID: \"f0d7e9e4-eeb8-42ac-aee1-db3a04662469\") " Oct 31 00:49:05.259023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.258991 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config" pod "f0d7e9e4-eeb8-42ac-aee1-db3a04662469" (UID: "f0d7e9e4-eeb8-42ac-aee1-db3a04662469"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.263801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.263760 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls" pod "f0d7e9e4-eeb8-42ac-aee1-db3a04662469" (UID: "f0d7e9e4-eeb8-42ac-aee1-db3a04662469"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.263905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.263819 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn" pod "f0d7e9e4-eeb8-42ac-aee1-db3a04662469" (UID: "f0d7e9e4-eeb8-42ac-aee1-db3a04662469"). InnerVolumeSpecName "kube-api-access-cnsbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.281773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.281745 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:05.281962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.281940 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="2572119d-0db7-4d48-8224-e32d9deba663" containerName="perses" containerID="cri-o://18c0567b9d9a863128c515f49ed02b3c4be0343c91b81d84aebba21d7011f700" gracePeriod=30 Oct 31 00:49:05.302310 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.302275 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.359080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359052 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\") pod \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " Oct 31 00:49:05.359234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359089 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\" (UniqueName: \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\") pod \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " Oct 31 00:49:05.359234 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359115 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\" (UniqueName: \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\") pod \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " Oct 31 00:49:05.359372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359282 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\") pod \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " Oct 31 00:49:05.359372 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359370 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\") pod \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\" (UID: \"6113840e-b89c-4c86-b33b-3d4cc6c1be47\") " Oct 31 00:49:05.359480 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359427 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca" pod "6113840e-b89c-4c86-b33b-3d4cc6c1be47" (UID: "6113840e-b89c-4c86-b33b-3d4cc6c1be47"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.359666 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359645 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.359954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359673 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnsbn\" (UniqueName: \"kubernetes.io/projected/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-kube-api-access-cnsbn\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.359954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359689 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f0d7e9e4-eeb8-42ac-aee1-db3a04662469-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.359954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359702 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.359954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359649 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config" pod "6113840e-b89c-4c86-b33b-3d4cc6c1be47" (UID: "6113840e-b89c-4c86-b33b-3d4cc6c1be47"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.359954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.359768 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca" pod "6113840e-b89c-4c86-b33b-3d4cc6c1be47" (UID: "6113840e-b89c-4c86-b33b-3d4cc6c1be47"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.361280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.361258 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls" pod "6113840e-b89c-4c86-b33b-3d4cc6c1be47" (UID: "6113840e-b89c-4c86-b33b-3d4cc6c1be47"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.361368 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.361284 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm" pod "6113840e-b89c-4c86-b33b-3d4cc6c1be47" (UID: "6113840e-b89c-4c86-b33b-3d4cc6c1be47"). InnerVolumeSpecName "kube-api-access-b2vqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444218 5577 generic.go:356] "Generic (PLEG): container finished" podID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" exitCode=0 Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444251 5577 generic.go:356] "Generic (PLEG): container finished" podID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" exitCode=0 Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444262 5577 generic.go:356] "Generic (PLEG): container finished" podID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" exitCode=0 Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444359 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerDied","Data":"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a"} Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444426 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerDied","Data":"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d"} Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444465 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerDied","Data":"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc"} Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444502 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" event={"ID":"f0d7e9e4-eeb8-42ac-aee1-db3a04662469","Type":"ContainerDied","Data":"f24df87493a035aaeb6e09741f644c18deb2585148c6c694587531c00abbcb6a"} Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444546 5577 scope.go:122] "RemoveContainer" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" Oct 31 00:49:05.446733 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.444695 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr" Oct 31 00:49:05.448531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.448393 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.459224 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.459197 5577 generic.go:356] "Generic (PLEG): container finished" podID="2d2ae77e-61af-4549-962d-14f168afaf50" containerID="38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826" exitCode=0 Oct 31 00:49:05.459466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.459276 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerDied","Data":"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826"} Oct 31 00:49:05.462773 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.460256 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2vqm\" (UniqueName: \"kubernetes.io/projected/6113840e-b89c-4c86-b33b-3d4cc6c1be47-kube-api-access-b2vqm\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.463160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.463054 5577 generic.go:356] "Generic (PLEG): container finished" podID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" containerID="cbe1f8b3df4ab495cd5d192c0c9cc7062277de97ff08771e3b5e04ef4751b30b" exitCode=0 Oct 31 00:49:05.463160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.463115 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"70db1a15-ce89-40a9-ae87-b1d2ce967f2d","Type":"ContainerDied","Data":"cbe1f8b3df4ab495cd5d192c0c9cc7062277de97ff08771e3b5e04ef4751b30b"} Oct 31 00:49:05.464612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.464534 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.464612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.464564 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/6113840e-b89c-4c86-b33b-3d4cc6c1be47-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.464612 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.464581 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6113840e-b89c-4c86-b33b-3d4cc6c1be47-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.464856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.464767 5577 generic.go:356] "Generic (PLEG): container finished" podID="2572119d-0db7-4d48-8224-e32d9deba663" containerID="18c0567b9d9a863128c515f49ed02b3c4be0343c91b81d84aebba21d7011f700" exitCode=0 Oct 31 00:49:05.464856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.464850 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"2572119d-0db7-4d48-8224-e32d9deba663","Type":"ContainerDied","Data":"18c0567b9d9a863128c515f49ed02b3c4be0343c91b81d84aebba21d7011f700"} Oct 31 00:49:05.471433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471414 5577 scope.go:122] "RemoveContainer" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" Oct 31 00:49:05.471600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471437 5577 generic.go:356] "Generic (PLEG): container finished" podID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" exitCode=0 Oct 31 00:49:05.471723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471691 5577 generic.go:356] "Generic (PLEG): container finished" podID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" exitCode=0 Oct 31 00:49:05.471828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471815 5577 generic.go:356] "Generic (PLEG): container finished" podID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" exitCode=0 Oct 31 00:49:05.471926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471458 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerDied","Data":"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9"} Oct 31 00:49:05.473741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.471532 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" Oct 31 00:49:05.473741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.472058 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerDied","Data":"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b"} Oct 31 00:49:05.473741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.472106 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerDied","Data":"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d"} Oct 31 00:49:05.473741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.472142 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl" event={"ID":"6113840e-b89c-4c86-b33b-3d4cc6c1be47","Type":"ContainerDied","Data":"566e0756dbd9125f52b88a8dee1b0f5d91420d26891ac597dad3d8bf59d1908b"} Oct 31 00:49:05.481447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.481424 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:49:05.484739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.484681 5577 scope.go:122] "RemoveContainer" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" Oct 31 00:49:05.486236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.486218 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-rrrlr"] Oct 31 00:49:05.497725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.497697 5577 scope.go:122] "RemoveContainer" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" Oct 31 00:49:05.501982 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.501960 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:49:05.502097 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.502070 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": container with ID starting with 68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a not found: ID does not exist" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" Oct 31 00:49:05.502165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.502103 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a"} err="failed to get container status \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": rpc error: code = NotFound desc = could not find container \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": container with ID starting with 68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a not found: ID does not exist" Oct 31 00:49:05.502165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.502128 5577 scope.go:122] "RemoveContainer" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" Oct 31 00:49:05.506991 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.506965 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": container with ID starting with 7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d not found: ID does not exist" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" Oct 31 00:49:05.507151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.507129 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d"} err="failed to get container status \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": rpc error: code = NotFound desc = could not find container \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": container with ID starting with 7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d not found: ID does not exist" Oct 31 00:49:05.507268 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.507256 5577 scope.go:122] "RemoveContainer" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" Oct 31 00:49:05.507378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.507358 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-7rszl"] Oct 31 00:49:05.507815 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.507796 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": container with ID starting with 6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc not found: ID does not exist" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" Oct 31 00:49:05.507904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.507821 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc"} err="failed to get container status \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": rpc error: code = NotFound desc = could not find container \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": container with ID starting with 6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc not found: ID does not exist" Oct 31 00:49:05.507904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.507843 5577 scope.go:122] "RemoveContainer" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" Oct 31 00:49:05.510245 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.510202 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a"} err="failed to get container status \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": rpc error: code = NotFound desc = could not find container \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": container with ID starting with 68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a not found: ID does not exist" Oct 31 00:49:05.510831 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.510811 5577 scope.go:122] "RemoveContainer" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" Oct 31 00:49:05.511357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.511340 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d"} err="failed to get container status \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": rpc error: code = NotFound desc = could not find container \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": container with ID starting with 7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d not found: ID does not exist" Oct 31 00:49:05.511510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.511500 5577 scope.go:122] "RemoveContainer" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" Oct 31 00:49:05.511940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.511873 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc"} err="failed to get container status \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": rpc error: code = NotFound desc = could not find container \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": container with ID starting with 6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc not found: ID does not exist" Oct 31 00:49:05.511940 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.511890 5577 scope.go:122] "RemoveContainer" containerID="68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a" Oct 31 00:49:05.512215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512149 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a"} err="failed to get container status \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": rpc error: code = NotFound desc = could not find container \"68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a\": container with ID starting with 68bce882d5aa5ce156b2de23e026024a7b21c16fc4aef82b40d9482a663e8b5a not found: ID does not exist" Oct 31 00:49:05.512215 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512166 5577 scope.go:122] "RemoveContainer" containerID="7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d" Oct 31 00:49:05.512440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512380 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d"} err="failed to get container status \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": rpc error: code = NotFound desc = could not find container \"7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d\": container with ID starting with 7207ee2092532d24b67a17c5cc909ba2fe15155f613d15601011c4ff71c9f50d not found: ID does not exist" Oct 31 00:49:05.512440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512397 5577 scope.go:122] "RemoveContainer" containerID="6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc" Oct 31 00:49:05.512640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512580 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc"} err="failed to get container status \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": rpc error: code = NotFound desc = could not find container \"6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc\": container with ID starting with 6b478a0a7ea4ce67485fe25b578eefadfa603f780fb068625d67e7e2b6ab2cfc not found: ID does not exist" Oct 31 00:49:05.512640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.512594 5577 scope.go:122] "RemoveContainer" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" Oct 31 00:49:05.523080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.523062 5577 scope.go:122] "RemoveContainer" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" Oct 31 00:49:05.533820 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.533800 5577 scope.go:122] "RemoveContainer" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" Oct 31 00:49:05.543863 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.543842 5577 scope.go:122] "RemoveContainer" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" Oct 31 00:49:05.544261 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.544240 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": container with ID starting with 0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9 not found: ID does not exist" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" Oct 31 00:49:05.544354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.544271 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9"} err="failed to get container status \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": rpc error: code = NotFound desc = could not find container \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": container with ID starting with 0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9 not found: ID does not exist" Oct 31 00:49:05.544354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.544295 5577 scope.go:122] "RemoveContainer" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" Oct 31 00:49:05.544663 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.544637 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": container with ID starting with 20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b not found: ID does not exist" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" Oct 31 00:49:05.544795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.544668 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b"} err="failed to get container status \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": rpc error: code = NotFound desc = could not find container \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": container with ID starting with 20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b not found: ID does not exist" Oct 31 00:49:05.544795 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.544691 5577 scope.go:122] "RemoveContainer" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" Oct 31 00:49:05.545108 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:05.545078 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": container with ID starting with d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d not found: ID does not exist" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" Oct 31 00:49:05.545194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545114 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d"} err="failed to get container status \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": rpc error: code = NotFound desc = could not find container \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": container with ID starting with d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d not found: ID does not exist" Oct 31 00:49:05.545194 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545134 5577 scope.go:122] "RemoveContainer" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" Oct 31 00:49:05.545391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545370 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9"} err="failed to get container status \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": rpc error: code = NotFound desc = could not find container \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": container with ID starting with 0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9 not found: ID does not exist" Oct 31 00:49:05.545457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545391 5577 scope.go:122] "RemoveContainer" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" Oct 31 00:49:05.545655 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545637 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b"} err="failed to get container status \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": rpc error: code = NotFound desc = could not find container \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": container with ID starting with 20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b not found: ID does not exist" Oct 31 00:49:05.545755 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545656 5577 scope.go:122] "RemoveContainer" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" Oct 31 00:49:05.545946 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545926 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d"} err="failed to get container status \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": rpc error: code = NotFound desc = could not find container \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": container with ID starting with d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d not found: ID does not exist" Oct 31 00:49:05.546016 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.545945 5577 scope.go:122] "RemoveContainer" containerID="0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9" Oct 31 00:49:05.546160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.546141 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9"} err="failed to get container status \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": rpc error: code = NotFound desc = could not find container \"0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9\": container with ID starting with 0bebc408ec4e1fc48979c4a2ed326cbd96aca00046694f90692a33c81974fba9 not found: ID does not exist" Oct 31 00:49:05.546229 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.546160 5577 scope.go:122] "RemoveContainer" containerID="20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b" Oct 31 00:49:05.546385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.546353 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b"} err="failed to get container status \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": rpc error: code = NotFound desc = could not find container \"20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b\": container with ID starting with 20f628f5b4fdbec8614409bb184c3a8037e74989117f9097ae9b10163f5e8d3b not found: ID does not exist" Oct 31 00:49:05.546460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.546386 5577 scope.go:122] "RemoveContainer" containerID="d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d" Oct 31 00:49:05.546620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.546592 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d"} err="failed to get container status \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": rpc error: code = NotFound desc = could not find container \"d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d\": container with ID starting with d1b703c925ec9bbd2854075077375d6078414cb0aa0b6d004c484b3215c78c2d not found: ID does not exist" Oct 31 00:49:05.580302 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.580275 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.591433 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.591411 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:05.596130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.596113 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:05.666466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666443 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.666551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666475 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.666586 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666562 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"2572119d-0db7-4d48-8224-e32d9deba663\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " Oct 31 00:49:05.666632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666587 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\" (UniqueName: \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\") pod \"2572119d-0db7-4d48-8224-e32d9deba663\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " Oct 31 00:49:05.666673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666639 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\" (UniqueName: \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\") pod \"2572119d-0db7-4d48-8224-e32d9deba663\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " Oct 31 00:49:05.666748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666682 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\" (UniqueName: \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\") pod \"2572119d-0db7-4d48-8224-e32d9deba663\" (UID: \"2572119d-0db7-4d48-8224-e32d9deba663\") " Oct 31 00:49:05.666748 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666723 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.666861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666760 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\" (UniqueName: \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.666861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666786 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.666861 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.666849 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\") pod \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\" (UID: \"70db1a15-ce89-40a9-ae87-b1d2ce967f2d\") " Oct 31 00:49:05.667053 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.667020 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config" pod "2572119d-0db7-4d48-8224-e32d9deba663" (UID: "2572119d-0db7-4d48-8224-e32d9deba663"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.667549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.667236 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2572119d-0db7-4d48-8224-e32d9deba663-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.668052 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.667977 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:05.669913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.669884 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "kube-api-access-2mdq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.669913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.669895 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.670078 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.670014 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg" pod "2572119d-0db7-4d48-8224-e32d9deba663" (UID: "2572119d-0db7-4d48-8224-e32d9deba663"). InnerVolumeSpecName "kube-api-access-km5lg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:05.670127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.670085 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.670390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.670365 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.670509 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.670449 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "2572119d-0db7-4d48-8224-e32d9deba663" (UID: "2572119d-0db7-4d48-8224-e32d9deba663"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:49:05.671261 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.671229 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs" pod "70db1a15-ce89-40a9-ae87-b1d2ce967f2d" (UID: "70db1a15-ce89-40a9-ae87-b1d2ce967f2d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:05.708306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.708270 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:05.768023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.767981 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:49:05.768023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768016 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km5lg\" (UniqueName: \"kubernetes.io/projected/2572119d-0db7-4d48-8224-e32d9deba663-kube-api-access-km5lg\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768026 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768035 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mdq9\" (UniqueName: \"kubernetes.io/projected/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-kube-api-access-2mdq9\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768044 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768054 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768062 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.768243 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.768070 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70db1a15-ce89-40a9-ae87-b1d2ce967f2d-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:05.782343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.782324 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:49:05.815153 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.815125 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins" pod "2572119d-0db7-4d48-8224-e32d9deba663" (UID: "2572119d-0db7-4d48-8224-e32d9deba663"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:05.868962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.868938 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:49:05.868962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:05.868961 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/2572119d-0db7-4d48-8224-e32d9deba663-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:06.476723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.476672 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"70db1a15-ce89-40a9-ae87-b1d2ce967f2d","Type":"ContainerDied","Data":"cabbfd52b41178e5c6c9cfd50382c0bb11b237415c18070a13f72300f2452118"} Oct 31 00:49:06.477325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.476679 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:06.477325 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.476757 5577 scope.go:122] "RemoveContainer" containerID="cbe1f8b3df4ab495cd5d192c0c9cc7062277de97ff08771e3b5e04ef4751b30b" Oct 31 00:49:06.478092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.477995 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:06.478092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.478041 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"2572119d-0db7-4d48-8224-e32d9deba663","Type":"ContainerDied","Data":"68ed043c5cfe5030ee9ab981050e00dd616a944a3dce8ed97c8468c44147a303"} Oct 31 00:49:06.486269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.486210 5577 scope.go:122] "RemoveContainer" containerID="18c0567b9d9a863128c515f49ed02b3c4be0343c91b81d84aebba21d7011f700" Oct 31 00:49:06.502176 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.502156 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:06.507081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.507060 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:06.518865 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.518846 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:06.523469 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:06.523449 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:07.128757 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:07.128727 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2572119d-0db7-4d48-8224-e32d9deba663" path="/var/lib/kubelet/pods/2572119d-0db7-4d48-8224-e32d9deba663/volumes" Oct 31 00:49:07.129113 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:07.129101 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" path="/var/lib/kubelet/pods/6113840e-b89c-4c86-b33b-3d4cc6c1be47/volumes" Oct 31 00:49:07.129514 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:07.129503 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" path="/var/lib/kubelet/pods/70db1a15-ce89-40a9-ae87-b1d2ce967f2d/volumes" Oct 31 00:49:07.130156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:07.130138 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" path="/var/lib/kubelet/pods/f0d7e9e4-eeb8-42ac-aee1-db3a04662469/volumes" Oct 31 00:49:07.282797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:07.282761 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.141:9093/-/ready\": dial tcp 10.133.0.141:9093: connect: connection refused" Oct 31 00:49:11.081862 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:11.081822 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:12.282611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:12.282577 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.141:9093/-/ready\": dial tcp 10.133.0.141:9093: connect: connection refused" Oct 31 00:49:15.293684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.293657 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:15.342436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342353 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342436 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342396 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342465 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342494 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342526 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342603 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342661 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342635 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\") pod \"2d2ae77e-61af-4549-962d-14f168afaf50\" (UID: \"2d2ae77e-61af-4549-962d-14f168afaf50\") " Oct 31 00:49:15.342963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.342835 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:15.344935 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.344908 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:15.345068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.345010 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "kube-api-access-hqjgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:15.345222 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.345197 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:15.345345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.345326 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:15.348735 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.348698 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:15.354956 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.354788 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config" pod "2d2ae77e-61af-4549-962d-14f168afaf50" (UID: "2d2ae77e-61af-4549-962d-14f168afaf50"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:15.444001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.443967 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.443996 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444001 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.444007 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2d2ae77e-61af-4549-962d-14f168afaf50-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.444018 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.444028 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2d2ae77e-61af-4549-962d-14f168afaf50-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.444037 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqjgw\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-kube-api-access-hqjgw\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.444221 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.444046 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2d2ae77e-61af-4549-962d-14f168afaf50-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.521116 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.521080 5577 generic.go:356] "Generic (PLEG): container finished" podID="2d2ae77e-61af-4549-962d-14f168afaf50" containerID="1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec" exitCode=0 Oct 31 00:49:15.521277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.521151 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:15.521277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.521170 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerDied","Data":"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec"} Oct 31 00:49:15.521277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.521232 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"2d2ae77e-61af-4549-962d-14f168afaf50","Type":"ContainerDied","Data":"2c67dd41ee71112c0f00ad31d06fb8d19578be23c49f087f750cc704ee7514dd"} Oct 31 00:49:15.521277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.521260 5577 scope.go:122] "RemoveContainer" containerID="38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826" Oct 31 00:49:15.530139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.530121 5577 scope.go:122] "RemoveContainer" containerID="1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec" Oct 31 00:49:15.537774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.537756 5577 scope.go:122] "RemoveContainer" containerID="61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6" Oct 31 00:49:15.544786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.544764 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:15.546092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546077 5577 scope.go:122] "RemoveContainer" containerID="38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826" Oct 31 00:49:15.546360 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:15.546339 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826\": container with ID starting with 38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826 not found: ID does not exist" containerID="38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826" Oct 31 00:49:15.546409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546370 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826"} err="failed to get container status \"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826\": rpc error: code = NotFound desc = could not find container \"38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826\": container with ID starting with 38b6cd2450b12d38785dfb65dbc1269db30f177f6053e5f986fa2df9bb8fb826 not found: ID does not exist" Oct 31 00:49:15.546409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546387 5577 scope.go:122] "RemoveContainer" containerID="1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec" Oct 31 00:49:15.546622 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:15.546605 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec\": container with ID starting with 1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec not found: ID does not exist" containerID="1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec" Oct 31 00:49:15.546693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546632 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec"} err="failed to get container status \"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec\": rpc error: code = NotFound desc = could not find container \"1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec\": container with ID starting with 1833e31b819774a8881764f27f22b989a1010f59685cb763f1c58c544968dfec not found: ID does not exist" Oct 31 00:49:15.546693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546653 5577 scope.go:122] "RemoveContainer" containerID="61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6" Oct 31 00:49:15.546923 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:15.546906 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6\": container with ID starting with 61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6 not found: ID does not exist" containerID="61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6" Oct 31 00:49:15.546984 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.546929 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6"} err="failed to get container status \"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6\": rpc error: code = NotFound desc = could not find container \"61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6\": container with ID starting with 61a0f3f3e4970b7998cac1509b6347ef86a824c94723f0be2d34a23599bb0ea6 not found: ID does not exist" Oct 31 00:49:15.548979 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:15.548959 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:17.060660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.060621 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.129818 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.129778 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" path="/var/lib/kubelet/pods/2d2ae77e-61af-4549-962d-14f168afaf50/volumes" Oct 31 00:49:17.178255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178226 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:49:17.178505 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178492 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2572119d-0db7-4d48-8224-e32d9deba663" containerName="perses" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178506 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2572119d-0db7-4d48-8224-e32d9deba663" containerName="perses" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178519 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178524 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178530 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="method-gate" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178535 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="method-gate" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178544 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="init-config-reloader" Oct 31 00:49:17.178549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178549 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="init-config-reloader" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178556 5577 cpu_manager.go:414] "containerMap: removing container" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="config-reloader" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178561 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="config-reloader" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178575 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178580 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178587 5577 cpu_manager.go:414] "containerMap: removing container" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178593 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178605 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178611 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178617 5577 cpu_manager.go:414] "containerMap: removing container" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178622 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178629 5577 cpu_manager.go:414] "containerMap: removing container" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="method-gate" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178633 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="method-gate" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178639 5577 cpu_manager.go:414] "containerMap: removing container" podUID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" containerName="otc-container" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178643 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" containerName="otc-container" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178682 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="alertmanager" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178690 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="method-gate" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178695 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d2ae77e-61af-4549-962d-14f168afaf50" containerName="config-reloader" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178702 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178723 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="prom-label-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178730 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d7e9e4-eeb8-42ac-aee1-db3a04662469" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178735 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="kube-rbac-proxy" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178740 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="6113840e-b89c-4c86-b33b-3d4cc6c1be47" containerName="method-gate" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178744 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="70db1a15-ce89-40a9-ae87-b1d2ce967f2d" containerName="otc-container" Oct 31 00:49:17.178805 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.178749 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="2572119d-0db7-4d48-8224-e32d9deba663" containerName="perses" Oct 31 00:49:17.182043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.182023 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.192270 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.192247 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:49:17.196840 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.196820 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:49:17.196969 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.196909 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.204373 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.204340 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.207420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.207400 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:49:17.233477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.233225 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:49:17.233477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.233296 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-hwlh4\"" Oct 31 00:49:17.233477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.233445 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-r4vw2\"" Oct 31 00:49:17.234285 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.234263 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:49:17.258627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258603 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nttbz\" (UniqueName: \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.258744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258644 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.258744 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258672 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.258839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258777 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.258839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258800 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.258914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258847 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.258914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258901 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.258993 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.258919 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxqj4\" (UniqueName: \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.264316 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264296 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:49:17.264416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264351 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:49:17.264416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264351 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:49:17.264416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264370 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:49:17.264416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264374 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:49:17.264416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.264399 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:49:17.334737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.334639 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.359679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359654 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nttbz\" (UniqueName: \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.359837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359691 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.359837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359737 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.359958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359897 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.359958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359929 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.360076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.359985 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.360076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360022 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.360076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360047 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxqj4\" (UniqueName: \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.360243 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:17.360157 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:49:17.360243 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:17.360212 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls podName:961d2643-213a-4cd6-8b2b-0719746a9131 nodeName:}" failed. No retries permitted until 2026-10-31 00:49:17.860192327 +0000 UTC m=+2996.851851903 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-n4bjk" (UID: "961d2643-213a-4cd6-8b2b-0719746a9131") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:49:17.360464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360441 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.360537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360514 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.360583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360538 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.360851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.360832 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.362688 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.362671 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.375890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.375864 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxqj4\" (UniqueName: \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-ctdcm\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.376017 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.375927 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nttbz\" (UniqueName: \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.473677 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.473639 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.499691 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.499660 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:49:17.699337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.699264 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:17.703887 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.703865 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.713152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.713072 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:17.719786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.719759 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:17.720922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.720901 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:17.723442 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.723422 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.738131 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.738104 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:17.763922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.763891 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.763922 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.763920 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.764166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.763942 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.764166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.763968 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.764166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.764078 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vtg7\" (UniqueName: \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.764166 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.764137 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs4dl\" (UniqueName: \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.764408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.764175 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.764408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.764210 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.764408 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.764273 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.782977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.782945 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:17.789450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.789426 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:49:17.792652 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:17.792626 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aff232_f1a9_4382_9619_d9375b5bfac2.slice/crio-729e4ac84ae1252dcc442e929b1dd21dc60d230ccec20497db23b368c1c3c69c WatchSource:0}: Error finding container 729e4ac84ae1252dcc442e929b1dd21dc60d230ccec20497db23b368c1c3c69c: Status 404 returned error can't find the container with id 729e4ac84ae1252dcc442e929b1dd21dc60d230ccec20497db23b368c1c3c69c Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799341 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799445 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799525 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799538 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cp86t\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799531 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799596 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:49:17.800022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.799648 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:17.865303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865247 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs4dl\" (UniqueName: \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865334 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.865430 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865371 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865435 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865466 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865542 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865535 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.865730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865559 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.865730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865590 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865617 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.865730 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.865687 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vtg7\" (UniqueName: \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.866060 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:17.866040 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:17.866128 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:17.866116 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs podName:ee9e4204-3f78-4536-9d6c-f4ff5d620c3d nodeName:}" failed. No retries permitted until 2026-10-31 00:49:18.366095139 +0000 UTC m=+2997.357754701 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:17.866199 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.866154 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.866672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.866651 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.866817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.866792 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.868423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.868399 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.869251 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.869155 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-n4bjk\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:17.871167 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.871143 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.871378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.871358 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.878679 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.878656 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs4dl\" (UniqueName: \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:17.878941 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.878924 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vtg7\" (UniqueName: \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:17.914239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:17.914207 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.045277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.045241 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.115495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.115469 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:49:18.180098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.180062 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.320145 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.320108 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:18.377668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.377633 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:18.377883 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:18.377866 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:18.377958 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:18.377938 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs podName:ee9e4204-3f78-4536-9d6c-f4ff5d620c3d nodeName:}" failed. No retries permitted until 2026-10-31 00:49:19.377922377 +0000 UTC m=+2998.369581943 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:49:18.393464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.393433 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:49:18.396204 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:18.396178 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961d2643_213a_4cd6_8b2b_0719746a9131.slice/crio-91d54755246db5f99a6adef81d35b71f29d70e4ebb492737897ccbbfee3f1f2d WatchSource:0}: Error finding container 91d54755246db5f99a6adef81d35b71f29d70e4ebb492737897ccbbfee3f1f2d: Status 404 returned error can't find the container with id 91d54755246db5f99a6adef81d35b71f29d70e4ebb492737897ccbbfee3f1f2d Oct 31 00:49:18.544266 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.544229 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerStarted","Data":"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f"} Oct 31 00:49:18.544380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.544339 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerStarted","Data":"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce"} Oct 31 00:49:18.544461 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.544379 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerStarted","Data":"91d54755246db5f99a6adef81d35b71f29d70e4ebb492737897ccbbfee3f1f2d"} Oct 31 00:49:18.546926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.546897 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerStarted","Data":"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089"} Oct 31 00:49:18.547020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.546975 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerStarted","Data":"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355"} Oct 31 00:49:18.547020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.547014 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerStarted","Data":"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452"} Oct 31 00:49:18.547099 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.547051 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerStarted","Data":"729e4ac84ae1252dcc442e929b1dd21dc60d230ccec20497db23b368c1c3c69c"} Oct 31 00:49:18.724766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:18.724702 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:19.387074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.387038 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:19.389589 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.389567 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:19.524801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.524772 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:19.553363 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.553332 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerStarted","Data":"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5"} Oct 31 00:49:19.575545 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.575491 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" podStartSLOduration=2.575474167 podStartE2EDuration="2.575474167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:18.568525529 +0000 UTC m=+2997.560185118" watchObservedRunningTime="2026-10-31 00:49:19.575474167 +0000 UTC m=+2998.567133753" Oct 31 00:49:19.576095 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.575860 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" podStartSLOduration=2.575848488 podStartE2EDuration="2.575848488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:19.574747439 +0000 UTC m=+2998.566407024" watchObservedRunningTime="2026-10-31 00:49:19.575848488 +0000 UTC m=+2998.567508070" Oct 31 00:49:19.801475 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.801400 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:19.804409 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:19.804380 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee9e4204_3f78_4536_9d6c_f4ff5d620c3d.slice/crio-a35802999cf9a3e7b83a06dc0759205284af8fe109e858d5880d16636a95e495 WatchSource:0}: Error finding container a35802999cf9a3e7b83a06dc0759205284af8fe109e858d5880d16636a95e495: Status 404 returned error can't find the container with id a35802999cf9a3e7b83a06dc0759205284af8fe109e858d5880d16636a95e495 Oct 31 00:49:19.806903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.806874 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:19.806980 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:19.806930 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:20.559968 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:20.559925 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d","Type":"ContainerStarted","Data":"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9"} Oct 31 00:49:20.560359 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:20.559985 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d","Type":"ContainerStarted","Data":"a35802999cf9a3e7b83a06dc0759205284af8fe109e858d5880d16636a95e495"} Oct 31 00:49:20.581175 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:20.581121 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=3.581103933 podStartE2EDuration="3.581103933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:20.580458198 +0000 UTC m=+2999.572117783" watchObservedRunningTime="2026-10-31 00:49:20.581103933 +0000 UTC m=+2999.572763517" Oct 31 00:49:21.425409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:21.425380 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:49:21.431184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:21.431163 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-t2vhd_99dfb869-c9bf-41d2-832a-d69ae57a802b/ovn-acl-logging/0.log" Oct 31 00:49:23.400250 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.400212 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.604162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.604128 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.727546 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.727467 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.818478 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.818443 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:23.823913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.823892 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.827859 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.827831 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:49:23.828379 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.828361 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:49:23.828576 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.828557 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:49:23.828649 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.828597 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:49:23.828690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.828658 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-wzs48\"" Oct 31 00:49:23.836191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.836172 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:23.864214 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.864180 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:23.927447 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927417 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927460 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927605 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927536 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927611 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927670 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8hp\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927766 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927739 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:23.927898 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:23.927778 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028492 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028636 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028686 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028759 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028737 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028762 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028802 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8hp\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.028999 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.028844 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.030406 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.029755 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.032126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.032103 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.032506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.032477 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.032506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.032492 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.032643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.032531 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.032643 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.032616 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.042949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.042922 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8hp\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.123096 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.123062 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:24.144178 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.144147 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:24.230816 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.230782 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.238425 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.238400 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.332423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.332333 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.332580 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.332489 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.337045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.337016 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.430348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.430316 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.436919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.436889 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:49:24.440433 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:24.440408 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5edc87_63a3_4560_8df9_a1030334c875.slice/crio-ca612441f434a0426452e109b4ef6c517aa4f151e7a240b5509f40d2ff4db1e3 WatchSource:0}: Error finding container ca612441f434a0426452e109b4ef6c517aa4f151e7a240b5509f40d2ff4db1e3: Status 404 returned error can't find the container with id ca612441f434a0426452e109b4ef6c517aa4f151e7a240b5509f40d2ff4db1e3 Oct 31 00:49:24.452466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.452441 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.578152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.578111 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerStarted","Data":"ca612441f434a0426452e109b4ef6c517aa4f151e7a240b5509f40d2ff4db1e3"} Oct 31 00:49:24.637732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.637629 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:49:24.645758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.645727 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:24.921429 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:24.921358 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:49:24.924455 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:24.924430 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod060b1e0b_7dfd_4168_8368_8fdd42f28ef0.slice/crio-65f66dbfb8e67680c1f017e98853f11a9f7b6a8ba340d0fe5ec78fc67bc3a2da WatchSource:0}: Error finding container 65f66dbfb8e67680c1f017e98853f11a9f7b6a8ba340d0fe5ec78fc67bc3a2da: Status 404 returned error can't find the container with id 65f66dbfb8e67680c1f017e98853f11a9f7b6a8ba340d0fe5ec78fc67bc3a2da Oct 31 00:49:25.583870 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:25.583831 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"060b1e0b-7dfd-4168-8368-8fdd42f28ef0","Type":"ContainerStarted","Data":"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866"} Oct 31 00:49:25.584385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:25.583883 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"060b1e0b-7dfd-4168-8368-8fdd42f28ef0","Type":"ContainerStarted","Data":"65f66dbfb8e67680c1f017e98853f11a9f7b6a8ba340d0fe5ec78fc67bc3a2da"} Oct 31 00:49:25.584385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:25.583945 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:25.585548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:25.585517 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerStarted","Data":"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db"} Oct 31 00:49:25.602903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:25.602855 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=8.602842079 podStartE2EDuration="8.602842079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:49:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:25.602548069 +0000 UTC m=+3004.594207654" watchObservedRunningTime="2026-10-31 00:49:25.602842079 +0000 UTC m=+3004.594501662" Oct 31 00:49:27.188435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:27.188394 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:29.293426 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:29.293375 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:49:29.605540 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:29.605456 5577 generic.go:356] "Generic (PLEG): container finished" podID="4a5edc87-63a3-4560-8df9-a1030334c875" containerID="c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db" exitCode=0 Oct 31 00:49:29.605700 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:29.605531 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerDied","Data":"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db"} Oct 31 00:49:30.612390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:30.612347 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerStarted","Data":"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b"} Oct 31 00:49:30.612813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:30.612427 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerStarted","Data":"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c"} Oct 31 00:49:30.612813 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:30.612647 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:30.641357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:30.641310 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.838953536 podStartE2EDuration="7.641296911s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.802343375s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:30.639427096 +0000 UTC m=+3009.631086725" watchObservedRunningTime="2026-10-31 00:49:30.641296911 +0000 UTC m=+3009.632956511" Oct 31 00:49:36.592455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:36.592427 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:49:36.617989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:36.617963 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:49:45.380817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:45.380776 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.598541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:45.598500 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:45.962552 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:45.962516 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:45.962776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:45.962726 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" containerName="otc-container" containerID="cri-o://21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9" gracePeriod=30 Oct 31 00:49:46.083717 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.083662 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.209518 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.209490 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.212788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.212690 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:49:46.325732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325677 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.325732 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325730 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.325933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325768 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\" (UniqueName: \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.325933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325818 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.325933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325840 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.325933 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.325918 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\") pod \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\" (UID: \"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d\") " Oct 31 00:49:46.328012 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.327976 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:46.329990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.329957 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:46.329990 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.329973 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:46.330136 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.330070 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:46.330483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.330457 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:46.330483 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.330472 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl" pod "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" (UID: "ee9e4204-3f78-4536-9d6c-f4ff5d620c3d"). InnerVolumeSpecName "kube-api-access-rs4dl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:46.426659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426621 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.426659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426653 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.426659 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426662 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.427149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426671 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs4dl\" (UniqueName: \"kubernetes.io/projected/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-kube-api-access-rs4dl\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.427149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426680 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.427149 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.426689 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:46.682109 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.682072 5577 generic.go:356] "Generic (PLEG): container finished" podID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" containerID="21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9" exitCode=0 Oct 31 00:49:46.682306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.682132 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.682306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.682165 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d","Type":"ContainerDied","Data":"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9"} Oct 31 00:49:46.682306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.682227 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ee9e4204-3f78-4536-9d6c-f4ff5d620c3d","Type":"ContainerDied","Data":"a35802999cf9a3e7b83a06dc0759205284af8fe109e858d5880d16636a95e495"} Oct 31 00:49:46.682306 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.682262 5577 scope.go:122] "RemoveContainer" containerID="21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9" Oct 31 00:49:46.691380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.691356 5577 scope.go:122] "RemoveContainer" containerID="21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9" Oct 31 00:49:46.691797 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:46.691635 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9\": container with ID starting with 21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9 not found: ID does not exist" containerID="21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9" Oct 31 00:49:46.691797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.691662 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9"} err="failed to get container status \"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9\": rpc error: code = NotFound desc = could not find container \"21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9\": container with ID starting with 21cbce7bb7ed5d5fc2cb935c985e2fa756b566e5374594c75a08d4754d070dc9 not found: ID does not exist" Oct 31 00:49:46.706072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.706052 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:46.710664 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.710637 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:46.737926 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.737902 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:46.738236 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.738224 5577 cpu_manager.go:414] "containerMap: removing container" podUID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" containerName="otc-container" Oct 31 00:49:46.738288 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.738237 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" containerName="otc-container" Oct 31 00:49:46.738329 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.738304 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" containerName="otc-container" Oct 31 00:49:46.742647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.742627 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.746977 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.746957 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:49:46.747081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.747068 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cp86t\"" Oct 31 00:49:46.747081 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.747074 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:49:46.747434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.747416 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:49:46.747434 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.747445 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:49:46.747597 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.747563 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:49:46.756033 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.756005 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:46.830376 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830341 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmnwd\" (UniqueName: \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.830572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830421 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.830572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830462 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.830572 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830557 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.830758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830625 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.830758 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.830656 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931036 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931091 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931123 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931164 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931231 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931394 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931279 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmnwd\" (UniqueName: \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.931924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.931894 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.934026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.933971 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.934026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.933993 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.934308 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.934287 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.934348 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.934323 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:46.943921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:46.943900 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmnwd\" (UniqueName: \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\") pod \"data-science-collector-collector-1\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:47.057927 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.057894 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:49:47.133575 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.133469 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee9e4204-3f78-4536-9d6c-f4ff5d620c3d" path="/var/lib/kubelet/pods/ee9e4204-3f78-4536-9d6c-f4ff5d620c3d/volumes" Oct 31 00:49:47.338064 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.338032 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:49:47.341932 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:49:47.341888 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250a0ede_4671_463d_84bb_70b60c1ac761.slice/crio-21a0b28468cf721d9aed30bbe34e0ec0f2d6e4dd64996e3588ead974fbc63d31 WatchSource:0}: Error finding container 21a0b28468cf721d9aed30bbe34e0ec0f2d6e4dd64996e3588ead974fbc63d31: Status 404 returned error can't find the container with id 21a0b28468cf721d9aed30bbe34e0ec0f2d6e4dd64996e3588ead974fbc63d31 Oct 31 00:49:47.344357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.344320 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:47.344457 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.344400 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:49:47.695633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.695545 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"250a0ede-4671-463d-84bb-70b60c1ac761","Type":"ContainerStarted","Data":"1301fe143c3fe194f6f2859e0050dffd258c8313efaf5f84cd9990ec0bcab260"} Oct 31 00:49:47.695633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.695619 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"250a0ede-4671-463d-84bb-70b60c1ac761","Type":"ContainerStarted","Data":"21a0b28468cf721d9aed30bbe34e0ec0f2d6e4dd64996e3588ead974fbc63d31"} Oct 31 00:49:47.717737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:49:47.717678 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.717665075 podStartE2EDuration="1.717665075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:47.716387363 +0000 UTC m=+3026.708046946" watchObservedRunningTime="2026-10-31 00:49:47.717665075 +0000 UTC m=+3026.709324658" Oct 31 00:49:59.292506 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:49:59.292465 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:50:03.040602 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.040570 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.164615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.164584 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.533060 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.533031 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:03.533275 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.533238 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="250a0ede-4671-463d-84bb-70b60c1ac761" containerName="otc-container" containerID="cri-o://1301fe143c3fe194f6f2859e0050dffd258c8313efaf5f84cd9990ec0bcab260" gracePeriod=30 Oct 31 00:50:03.651849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.651811 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:03.775010 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:03.774974 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:05.776179 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.776150 5577 generic.go:356] "Generic (PLEG): container finished" podID="250a0ede-4671-463d-84bb-70b60c1ac761" containerID="1301fe143c3fe194f6f2859e0050dffd258c8313efaf5f84cd9990ec0bcab260" exitCode=0 Oct 31 00:50:05.776517 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.776185 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"250a0ede-4671-463d-84bb-70b60c1ac761","Type":"ContainerDied","Data":"1301fe143c3fe194f6f2859e0050dffd258c8313efaf5f84cd9990ec0bcab260"} Oct 31 00:50:05.885530 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.885504 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:05.993949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.993865 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.993949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.993920 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\" (UniqueName: \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.994189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.993955 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.994189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.993991 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.994189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.994033 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.994189 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.994069 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\" (UniqueName: \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\") pod \"250a0ede-4671-463d-84bb-70b60c1ac761\" (UID: \"250a0ede-4671-463d-84bb-70b60c1ac761\") " Oct 31 00:50:05.994449 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.994417 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:05.996440 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.996404 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:05.996962 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.996935 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:05.997068 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.997042 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:05.997132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.997073 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "kube-api-access-lmnwd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:05.997132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:05.997071 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs" pod "250a0ede-4671-463d-84bb-70b60c1ac761" (UID: "250a0ede-4671-463d-84bb-70b60c1ac761"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:06.095072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095041 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmnwd\" (UniqueName: \"kubernetes.io/projected/250a0ede-4671-463d-84bb-70b60c1ac761-kube-api-access-lmnwd\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.095072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095066 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.095072 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095075 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/250a0ede-4671-463d-84bb-70b60c1ac761-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.095301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095086 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.095301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095095 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.095301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.095103 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/250a0ede-4671-463d-84bb-70b60c1ac761-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:06.781584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.781555 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:06.781584 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.781564 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"250a0ede-4671-463d-84bb-70b60c1ac761","Type":"ContainerDied","Data":"21a0b28468cf721d9aed30bbe34e0ec0f2d6e4dd64996e3588ead974fbc63d31"} Oct 31 00:50:06.782173 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.781619 5577 scope.go:122] "RemoveContainer" containerID="1301fe143c3fe194f6f2859e0050dffd258c8313efaf5f84cd9990ec0bcab260" Oct 31 00:50:06.803800 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.803777 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:06.808367 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.808344 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:06.836533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.836505 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:06.836879 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.836864 5577 cpu_manager.go:414] "containerMap: removing container" podUID="250a0ede-4671-463d-84bb-70b60c1ac761" containerName="otc-container" Oct 31 00:50:06.836942 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.836880 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250a0ede-4671-463d-84bb-70b60c1ac761" containerName="otc-container" Oct 31 00:50:06.836986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.836951 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="250a0ede-4671-463d-84bb-70b60c1ac761" containerName="otc-container" Oct 31 00:50:06.841182 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.841168 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:06.845264 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.845238 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:50:06.845503 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.845487 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:50:06.846162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.846145 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:50:06.846162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.846153 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:50:06.846328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.846312 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:50:06.846328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.846313 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cp86t\"" Oct 31 00:50:06.851841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:06.851822 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:07.002955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.002923 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46p44\" (UniqueName: \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.002955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.002966 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.003195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.002997 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.003195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.003108 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.003195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.003135 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.003195 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.003174 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104448 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104485 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104529 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104512 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104541 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46p44\" (UniqueName: \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104570 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.104847 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.104597 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.105338 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.105309 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.107232 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.107212 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.107328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.107269 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.107631 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.107611 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.107673 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.107619 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.117559 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.117538 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46p44\" (UniqueName: \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\") pod \"data-science-collector-collector-1\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.129486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.129460 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250a0ede-4671-463d-84bb-70b60c1ac761" path="/var/lib/kubelet/pods/250a0ede-4671-463d-84bb-70b60c1ac761/volumes" Oct 31 00:50:07.153776 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.153752 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:07.453104 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.453031 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:07.456360 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:50:07.456332 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90120372_c1e7_4e29_9c9b_4591e8d48969.slice/crio-4e799d8f262eddee932508552968ccc4e08318198a19741bc43353484bf4d459 WatchSource:0}: Error finding container 4e799d8f262eddee932508552968ccc4e08318198a19741bc43353484bf4d459: Status 404 returned error can't find the container with id 4e799d8f262eddee932508552968ccc4e08318198a19741bc43353484bf4d459 Oct 31 00:50:07.458411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.458383 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:07.458471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.458442 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:07.787671 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.787628 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"90120372-c1e7-4e29-9c9b-4591e8d48969","Type":"ContainerStarted","Data":"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3"} Oct 31 00:50:07.788156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.787721 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"90120372-c1e7-4e29-9c9b-4591e8d48969","Type":"ContainerStarted","Data":"4e799d8f262eddee932508552968ccc4e08318198a19741bc43353484bf4d459"} Oct 31 00:50:07.809385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:07.809313 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.809296728 podStartE2EDuration="1.809296728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:07.80850994 +0000 UTC m=+3046.800169527" watchObservedRunningTime="2026-10-31 00:50:07.809296728 +0000 UTC m=+3046.800956312" Oct 31 00:50:09.282658 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.282617 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:50:09.283165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.282921 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="kube-rbac-proxy" containerID="cri-o://21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" gracePeriod=30 Oct 31 00:50:09.283165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.283000 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="prom-label-proxy" containerID="cri-o://34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" gracePeriod=30 Oct 31 00:50:09.283165 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.283025 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="method-gate" containerID="cri-o://c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" gracePeriod=30 Oct 31 00:50:09.304218 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.304193 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:50:09.304512 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.304486 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="kube-rbac-proxy" containerID="cri-o://a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" gracePeriod=30 Oct 31 00:50:09.304621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.304513 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="method-gate" containerID="cri-o://9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" gracePeriod=30 Oct 31 00:50:09.304621 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.304487 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="prom-label-proxy" containerID="cri-o://f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" gracePeriod=30 Oct 31 00:50:09.511047 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.509896 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.525045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.525016 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:09.525261 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.525235 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" containerID="cri-o://ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c" gracePeriod=120 Oct 31 00:50:09.525341 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.525262 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="config-reloader" containerID="cri-o://5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b" gracePeriod=120 Oct 31 00:50:09.644257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.644173 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.645385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.645360 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:50:09.648049 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.648027 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:50:09.665395 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.665367 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:09.727452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727406 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\" (UniqueName: \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\") pod \"f2aff232-f1a9-4382-9619-d9375b5bfac2\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " Oct 31 00:50:09.727452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727453 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") pod \"961d2643-213a-4cd6-8b2b-0719746a9131\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727489 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\" (UniqueName: \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\") pod \"961d2643-213a-4cd6-8b2b-0719746a9131\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727524 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\") pod \"f2aff232-f1a9-4382-9619-d9375b5bfac2\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727563 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\") pod \"f2aff232-f1a9-4382-9619-d9375b5bfac2\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727588 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\") pod \"f2aff232-f1a9-4382-9619-d9375b5bfac2\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727629 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\" (UniqueName: \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\") pod \"f2aff232-f1a9-4382-9619-d9375b5bfac2\" (UID: \"f2aff232-f1a9-4382-9619-d9375b5bfac2\") " Oct 31 00:50:09.727763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.727703 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\" (UniqueName: \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\") pod \"961d2643-213a-4cd6-8b2b-0719746a9131\" (UID: \"961d2643-213a-4cd6-8b2b-0719746a9131\") " Oct 31 00:50:09.728196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.728149 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config" pod "f2aff232-f1a9-4382-9619-d9375b5bfac2" (UID: "f2aff232-f1a9-4382-9619-d9375b5bfac2"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.728196 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.728153 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca" pod "f2aff232-f1a9-4382-9619-d9375b5bfac2" (UID: "f2aff232-f1a9-4382-9619-d9375b5bfac2"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.728328 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.728201 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config" pod "961d2643-213a-4cd6-8b2b-0719746a9131" (UID: "961d2643-213a-4cd6-8b2b-0719746a9131"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.729391 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.729362 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca" pod "f2aff232-f1a9-4382-9619-d9375b5bfac2" (UID: "f2aff232-f1a9-4382-9619-d9375b5bfac2"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:09.730815 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.730781 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz" pod "961d2643-213a-4cd6-8b2b-0719746a9131" (UID: "961d2643-213a-4cd6-8b2b-0719746a9131"). InnerVolumeSpecName "kube-api-access-nttbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.730944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.730856 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls" pod "f2aff232-f1a9-4382-9619-d9375b5bfac2" (UID: "f2aff232-f1a9-4382-9619-d9375b5bfac2"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.731187 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.731168 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4" pod "f2aff232-f1a9-4382-9619-d9375b5bfac2" (UID: "f2aff232-f1a9-4382-9619-d9375b5bfac2"). InnerVolumeSpecName "kube-api-access-bxqj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:09.731598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.731576 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls" pod "961d2643-213a-4cd6-8b2b-0719746a9131" (UID: "961d2643-213a-4cd6-8b2b-0719746a9131"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:09.742851 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.742825 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:09.743028 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.742991 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" containerName="perses" containerID="cri-o://ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866" gracePeriod=30 Oct 31 00:50:09.782921 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.782886 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.800126 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.800054 5577 generic.go:356] "Generic (PLEG): container finished" podID="4a5edc87-63a3-4560-8df9-a1030334c875" containerID="5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b" exitCode=0 Oct 31 00:50:09.800267 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.800124 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerDied","Data":"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b"} Oct 31 00:50:09.802150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802126 5577 generic.go:356] "Generic (PLEG): container finished" podID="961d2643-213a-4cd6-8b2b-0719746a9131" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" exitCode=0 Oct 31 00:50:09.802150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802149 5577 generic.go:356] "Generic (PLEG): container finished" podID="961d2643-213a-4cd6-8b2b-0719746a9131" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" exitCode=0 Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802158 5577 generic.go:356] "Generic (PLEG): container finished" podID="961d2643-213a-4cd6-8b2b-0719746a9131" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" exitCode=0 Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802190 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802200 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerDied","Data":"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5"} Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802262 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerDied","Data":"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f"} Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802301 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerDied","Data":"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce"} Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802338 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk" event={"ID":"961d2643-213a-4cd6-8b2b-0719746a9131","Type":"ContainerDied","Data":"91d54755246db5f99a6adef81d35b71f29d70e4ebb492737897ccbbfee3f1f2d"} Oct 31 00:50:09.802347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.802341 5577 scope.go:122] "RemoveContainer" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" Oct 31 00:50:09.804280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804262 5577 generic.go:356] "Generic (PLEG): container finished" podID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" exitCode=0 Oct 31 00:50:09.804280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804281 5577 generic.go:356] "Generic (PLEG): container finished" podID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" exitCode=0 Oct 31 00:50:09.804450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804287 5577 generic.go:356] "Generic (PLEG): container finished" podID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" exitCode=0 Oct 31 00:50:09.804450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804324 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" Oct 31 00:50:09.804450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804351 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerDied","Data":"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089"} Oct 31 00:50:09.804450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804419 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerDied","Data":"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355"} Oct 31 00:50:09.804660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804465 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerDied","Data":"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452"} Oct 31 00:50:09.804660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804505 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm" event={"ID":"f2aff232-f1a9-4382-9619-d9375b5bfac2","Type":"ContainerDied","Data":"729e4ac84ae1252dcc442e929b1dd21dc60d230ccec20497db23b368c1c3c69c"} Oct 31 00:50:09.804792 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.804769 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="90120372-c1e7-4e29-9c9b-4591e8d48969" containerName="otc-container" containerID="cri-o://48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3" gracePeriod=30 Oct 31 00:50:09.828508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828480 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f2aff232-f1a9-4382-9619-d9375b5bfac2-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828502 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/961d2643-213a-4cd6-8b2b-0719746a9131-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828508 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828511 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nttbz\" (UniqueName: \"kubernetes.io/projected/961d2643-213a-4cd6-8b2b-0719746a9131-kube-api-access-nttbz\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828521 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828532 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828539 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aff232-f1a9-4382-9619-d9375b5bfac2-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828547 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxqj4\" (UniqueName: \"kubernetes.io/projected/f2aff232-f1a9-4382-9619-d9375b5bfac2-kube-api-access-bxqj4\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.828740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.828555 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/961d2643-213a-4cd6-8b2b-0719746a9131-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:09.848537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.848505 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:50:09.852944 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.852919 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-n4bjk"] Oct 31 00:50:09.869144 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.869114 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:50:09.875400 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.875374 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-ctdcm"] Oct 31 00:50:09.893587 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.893565 5577 scope.go:122] "RemoveContainer" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" Oct 31 00:50:09.918283 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.918262 5577 scope.go:122] "RemoveContainer" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" Oct 31 00:50:09.923336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.923238 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:09.930743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.930697 5577 scope.go:122] "RemoveContainer" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" Oct 31 00:50:09.931113 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.931083 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": container with ID starting with f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5 not found: ID does not exist" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" Oct 31 00:50:09.931238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.931123 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5"} err="failed to get container status \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": rpc error: code = NotFound desc = could not find container \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": container with ID starting with f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5 not found: ID does not exist" Oct 31 00:50:09.931238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.931146 5577 scope.go:122] "RemoveContainer" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" Oct 31 00:50:09.931499 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.931474 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": container with ID starting with 9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f not found: ID does not exist" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" Oct 31 00:50:09.931598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.931510 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f"} err="failed to get container status \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": rpc error: code = NotFound desc = could not find container \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": container with ID starting with 9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f not found: ID does not exist" Oct 31 00:50:09.931598 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.931532 5577 scope.go:122] "RemoveContainer" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" Oct 31 00:50:09.932062 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.932042 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": container with ID starting with a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce not found: ID does not exist" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" Oct 31 00:50:09.932172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932067 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce"} err="failed to get container status \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": rpc error: code = NotFound desc = could not find container \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": container with ID starting with a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce not found: ID does not exist" Oct 31 00:50:09.932172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932088 5577 scope.go:122] "RemoveContainer" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" Oct 31 00:50:09.932390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932368 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5"} err="failed to get container status \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": rpc error: code = NotFound desc = could not find container \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": container with ID starting with f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5 not found: ID does not exist" Oct 31 00:50:09.932390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932389 5577 scope.go:122] "RemoveContainer" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" Oct 31 00:50:09.932699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932678 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f"} err="failed to get container status \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": rpc error: code = NotFound desc = could not find container \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": container with ID starting with 9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f not found: ID does not exist" Oct 31 00:50:09.932699 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.932700 5577 scope.go:122] "RemoveContainer" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" Oct 31 00:50:09.933098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933078 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce"} err="failed to get container status \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": rpc error: code = NotFound desc = could not find container \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": container with ID starting with a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce not found: ID does not exist" Oct 31 00:50:09.933172 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933100 5577 scope.go:122] "RemoveContainer" containerID="f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5" Oct 31 00:50:09.933458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933335 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5"} err="failed to get container status \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": rpc error: code = NotFound desc = could not find container \"f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5\": container with ID starting with f2aacdcba4275e56561c88ef596504d0576502f16a98f88aca1b4d3fa247e6a5 not found: ID does not exist" Oct 31 00:50:09.933458 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933351 5577 scope.go:122] "RemoveContainer" containerID="9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f" Oct 31 00:50:09.933934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933898 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f"} err="failed to get container status \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": rpc error: code = NotFound desc = could not find container \"9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f\": container with ID starting with 9841005f1b81869e0faf137eb84e1d913458fe824b72f072e0b22e587cc22b7f not found: ID does not exist" Oct 31 00:50:09.933934 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.933932 5577 scope.go:122] "RemoveContainer" containerID="a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce" Oct 31 00:50:09.934277 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.934247 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce"} err="failed to get container status \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": rpc error: code = NotFound desc = could not find container \"a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce\": container with ID starting with a20a02e19a160a1265839c6aa183aa3d175dd8d6675d781c19b20bf31efe87ce not found: ID does not exist" Oct 31 00:50:09.934366 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.934277 5577 scope.go:122] "RemoveContainer" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" Oct 31 00:50:09.948254 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.948227 5577 scope.go:122] "RemoveContainer" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" Oct 31 00:50:09.961750 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.961427 5577 scope.go:122] "RemoveContainer" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" Oct 31 00:50:09.973279 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.973220 5577 scope.go:122] "RemoveContainer" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" Oct 31 00:50:09.973676 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.973651 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": container with ID starting with 34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089 not found: ID does not exist" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" Oct 31 00:50:09.973893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.973688 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089"} err="failed to get container status \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": rpc error: code = NotFound desc = could not find container \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": container with ID starting with 34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089 not found: ID does not exist" Oct 31 00:50:09.973976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.973901 5577 scope.go:122] "RemoveContainer" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" Oct 31 00:50:09.974270 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.974244 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": container with ID starting with c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355 not found: ID does not exist" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" Oct 31 00:50:09.974336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974278 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355"} err="failed to get container status \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": rpc error: code = NotFound desc = could not find container \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": container with ID starting with c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355 not found: ID does not exist" Oct 31 00:50:09.974336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974319 5577 scope.go:122] "RemoveContainer" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" Oct 31 00:50:09.974610 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:09.974590 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": container with ID starting with 21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452 not found: ID does not exist" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" Oct 31 00:50:09.974697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974618 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452"} err="failed to get container status \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": rpc error: code = NotFound desc = could not find container \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": container with ID starting with 21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452 not found: ID does not exist" Oct 31 00:50:09.974697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974638 5577 scope.go:122] "RemoveContainer" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" Oct 31 00:50:09.974963 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974940 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089"} err="failed to get container status \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": rpc error: code = NotFound desc = could not find container \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": container with ID starting with 34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089 not found: ID does not exist" Oct 31 00:50:09.975056 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.974964 5577 scope.go:122] "RemoveContainer" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" Oct 31 00:50:09.975257 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975232 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355"} err="failed to get container status \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": rpc error: code = NotFound desc = could not find container \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": container with ID starting with c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355 not found: ID does not exist" Oct 31 00:50:09.975332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975257 5577 scope.go:122] "RemoveContainer" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" Oct 31 00:50:09.975533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975511 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452"} err="failed to get container status \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": rpc error: code = NotFound desc = could not find container \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": container with ID starting with 21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452 not found: ID does not exist" Oct 31 00:50:09.975616 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975533 5577 scope.go:122] "RemoveContainer" containerID="34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089" Oct 31 00:50:09.975839 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975816 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089"} err="failed to get container status \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": rpc error: code = NotFound desc = could not find container \"34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089\": container with ID starting with 34e46fbc137d6dc7fe001e956a01646c931cb2d2940bc44f10b79a94d6de6089 not found: ID does not exist" Oct 31 00:50:09.975936 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.975839 5577 scope.go:122] "RemoveContainer" containerID="c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355" Oct 31 00:50:09.976155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.976136 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355"} err="failed to get container status \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": rpc error: code = NotFound desc = could not find container \"c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355\": container with ID starting with c181a140913cf5844a0d4f25af92be5a31484fa53e8a66d0a65be784da7eb355 not found: ID does not exist" Oct 31 00:50:09.976239 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.976157 5577 scope.go:122] "RemoveContainer" containerID="21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452" Oct 31 00:50:09.976464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:09.976443 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452"} err="failed to get container status \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": rpc error: code = NotFound desc = could not find container \"21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452\": container with ID starting with 21cdd2cc617a065fc87c80483da1e37dab04c1f521659d53574178ffad512452 not found: ID does not exist" Oct 31 00:50:10.060327 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.060237 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:10.085101 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.085077 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:10.094361 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.094331 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:10.131387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131358 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\" (UniqueName: \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\") pod \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " Oct 31 00:50:10.131387 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131393 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\" (UniqueName: \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\") pod \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131419 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131464 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\" (UniqueName: \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\") pod \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131485 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131535 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\" (UniqueName: \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131567 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131620 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131602 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\" (UniqueName: \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131743 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\" (UID: \"060b1e0b-7dfd-4168-8368-8fdd42f28ef0\") " Oct 31 00:50:10.131967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131778 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\") pod \"90120372-c1e7-4e29-9c9b-4591e8d48969\" (UID: \"90120372-c1e7-4e29-9c9b-4591e8d48969\") " Oct 31 00:50:10.131967 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.131932 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config" pod "060b1e0b-7dfd-4168-8368-8fdd42f28ef0" (UID: "060b1e0b-7dfd-4168-8368-8fdd42f28ef0"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.132160 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.132059 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.132856 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.132596 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.135819 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.135774 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.136080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.136003 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "kube-api-access-46p44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.136295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.136211 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.136632 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.136601 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "060b1e0b-7dfd-4168-8368-8fdd42f28ef0" (UID: "060b1e0b-7dfd-4168-8368-8fdd42f28ef0"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:10.137125 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.136949 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.137640 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.137611 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7" pod "060b1e0b-7dfd-4168-8368-8fdd42f28ef0" (UID: "060b1e0b-7dfd-4168-8368-8fdd42f28ef0"). InnerVolumeSpecName "kube-api-access-4vtg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.138084 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.138067 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs" pod "90120372-c1e7-4e29-9c9b-4591e8d48969" (UID: "90120372-c1e7-4e29-9c9b-4591e8d48969"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.232761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232698 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vtg7\" (UniqueName: \"kubernetes.io/projected/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-kube-api-access-4vtg7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.232761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232752 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.232761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232761 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/90120372-c1e7-4e29-9c9b-4591e8d48969-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.232761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232771 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.233132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232780 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46p44\" (UniqueName: \"kubernetes.io/projected/90120372-c1e7-4e29-9c9b-4591e8d48969-kube-api-access-46p44\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.233132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232811 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:50:10.233132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232821 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.233132 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.232830 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/90120372-c1e7-4e29-9c9b-4591e8d48969-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.247477 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.247452 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:50:10.285418 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.285382 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins" pod "060b1e0b-7dfd-4168-8368-8fdd42f28ef0" (UID: "060b1e0b-7dfd-4168-8368-8fdd42f28ef0"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:10.334253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.334178 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:50:10.334253 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.334206 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/060b1e0b-7dfd-4168-8368-8fdd42f28ef0-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.810837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.810806 5577 generic.go:356] "Generic (PLEG): container finished" podID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" containerID="ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866" exitCode=0 Oct 31 00:50:10.811044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.810866 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:10.811044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.810891 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"060b1e0b-7dfd-4168-8368-8fdd42f28ef0","Type":"ContainerDied","Data":"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866"} Oct 31 00:50:10.811044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.810944 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"060b1e0b-7dfd-4168-8368-8fdd42f28ef0","Type":"ContainerDied","Data":"65f66dbfb8e67680c1f017e98853f11a9f7b6a8ba340d0fe5ec78fc67bc3a2da"} Oct 31 00:50:10.811044 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.810967 5577 scope.go:122] "RemoveContainer" containerID="ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866" Oct 31 00:50:10.813276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.813250 5577 generic.go:356] "Generic (PLEG): container finished" podID="90120372-c1e7-4e29-9c9b-4591e8d48969" containerID="48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3" exitCode=0 Oct 31 00:50:10.813390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.813297 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:10.813390 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.813337 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"90120372-c1e7-4e29-9c9b-4591e8d48969","Type":"ContainerDied","Data":"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3"} Oct 31 00:50:10.813507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.813389 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"90120372-c1e7-4e29-9c9b-4591e8d48969","Type":"ContainerDied","Data":"4e799d8f262eddee932508552968ccc4e08318198a19741bc43353484bf4d459"} Oct 31 00:50:10.820371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.820353 5577 scope.go:122] "RemoveContainer" containerID="ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866" Oct 31 00:50:10.820645 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:10.820626 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866\": container with ID starting with ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866 not found: ID does not exist" containerID="ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866" Oct 31 00:50:10.820754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.820651 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866"} err="failed to get container status \"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866\": rpc error: code = NotFound desc = could not find container \"ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866\": container with ID starting with ed53383c62c112c447306318d8efc3df9ee4ff2f7e846292e835448bdd93e866 not found: ID does not exist" Oct 31 00:50:10.820754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.820667 5577 scope.go:122] "RemoveContainer" containerID="48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3" Oct 31 00:50:10.828063 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.828046 5577 scope.go:122] "RemoveContainer" containerID="48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3" Oct 31 00:50:10.828357 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:10.828339 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3\": container with ID starting with 48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3 not found: ID does not exist" containerID="48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3" Oct 31 00:50:10.828409 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.828364 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3"} err="failed to get container status \"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3\": rpc error: code = NotFound desc = could not find container \"48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3\": container with ID starting with 48bbdd2b275e507ceab67e5fae2f52c7d7e07c9dc3822a4e639a6721f4bdbee3 not found: ID does not exist" Oct 31 00:50:10.833162 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.833141 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:10.839498 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.839477 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:10.851139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.851118 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:10.861141 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:10.857098 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:11.129725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:11.129623 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" path="/var/lib/kubelet/pods/060b1e0b-7dfd-4168-8368-8fdd42f28ef0/volumes" Oct 31 00:50:11.130102 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:11.130083 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90120372-c1e7-4e29-9c9b-4591e8d48969" path="/var/lib/kubelet/pods/90120372-c1e7-4e29-9c9b-4591e8d48969/volumes" Oct 31 00:50:11.130467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:11.130451 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" path="/var/lib/kubelet/pods/961d2643-213a-4cd6-8b2b-0719746a9131/volumes" Oct 31 00:50:11.131123 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:11.131100 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" path="/var/lib/kubelet/pods/f2aff232-f1a9-4382-9619-d9375b5bfac2/volumes" Oct 31 00:50:11.617398 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:11.617351 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.148:9093/-/ready\": dial tcp 10.133.0.148:9093: connect: connection refused" Oct 31 00:50:16.617260 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:16.617220 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.148:9093/-/ready\": dial tcp 10.133.0.148:9093: connect: connection refused" Oct 31 00:50:19.771076 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.771047 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:19.804151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804070 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804117 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804151 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804146 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804192 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804236 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804276 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.804435 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.804338 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\") pod \"4a5edc87-63a3-4560-8df9-a1030334c875\" (UID: \"4a5edc87-63a3-4560-8df9-a1030334c875\") " Oct 31 00:50:19.805128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.805055 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:19.807092 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.807051 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:19.807498 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.807467 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:19.807788 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.807768 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "kube-api-access-pf8hp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:19.808139 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.808118 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:19.812150 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.812125 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:19.819782 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.819754 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config" pod "4a5edc87-63a3-4560-8df9-a1030334c875" (UID: "4a5edc87-63a3-4560-8df9-a1030334c875"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:19.861455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.861420 5577 generic.go:356] "Generic (PLEG): container finished" podID="4a5edc87-63a3-4560-8df9-a1030334c875" containerID="ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c" exitCode=0 Oct 31 00:50:19.861634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.861467 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerDied","Data":"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c"} Oct 31 00:50:19.861634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.861547 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:19.861634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.861572 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"4a5edc87-63a3-4560-8df9-a1030334c875","Type":"ContainerDied","Data":"ca612441f434a0426452e109b4ef6c517aa4f151e7a240b5509f40d2ff4db1e3"} Oct 31 00:50:19.861634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.861610 5577 scope.go:122] "RemoveContainer" containerID="5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b" Oct 31 00:50:19.874002 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.873979 5577 scope.go:122] "RemoveContainer" containerID="ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c" Oct 31 00:50:19.884093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.884067 5577 scope.go:122] "RemoveContainer" containerID="c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db" Oct 31 00:50:19.890291 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.890266 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:19.892465 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.892440 5577 scope.go:122] "RemoveContainer" containerID="5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b" Oct 31 00:50:19.892815 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:19.892788 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b\": container with ID starting with 5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b not found: ID does not exist" containerID="5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b" Oct 31 00:50:19.892947 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.892923 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b"} err="failed to get container status \"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b\": rpc error: code = NotFound desc = could not find container \"5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b\": container with ID starting with 5084176105ede19aa137deac76de93905e69534ad3213062fbf771e374ee250b not found: ID does not exist" Oct 31 00:50:19.893046 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.892947 5577 scope.go:122] "RemoveContainer" containerID="ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c" Oct 31 00:50:19.893266 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:19.893235 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c\": container with ID starting with ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c not found: ID does not exist" containerID="ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c" Oct 31 00:50:19.893364 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.893277 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c"} err="failed to get container status \"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c\": rpc error: code = NotFound desc = could not find container \"ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c\": container with ID starting with ad2de5d4a16e079736f5ecf998ac904e9aa21d1e1ed41b0200b678b4c412d64c not found: ID does not exist" Oct 31 00:50:19.893364 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.893300 5577 scope.go:122] "RemoveContainer" containerID="c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db" Oct 31 00:50:19.893639 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:19.893621 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db\": container with ID starting with c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db not found: ID does not exist" containerID="c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db" Oct 31 00:50:19.893693 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.893649 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db"} err="failed to get container status \"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db\": rpc error: code = NotFound desc = could not find container \"c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db\": container with ID starting with c17a7f2050264c301abeed6d779b172ecd46d9192518b31fa5b6fd62be5395db not found: ID does not exist" Oct 31 00:50:19.897599 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.897573 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:19.905026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905002 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905026 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905036 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905045 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf8hp\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-kube-api-access-pf8hp\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905076 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4a5edc87-63a3-4560-8df9-a1030334c875-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905085 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4a5edc87-63a3-4560-8df9-a1030334c875-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.905188 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.905095 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/4a5edc87-63a3-4560-8df9-a1030334c875-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:19.949067 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:19.949027 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.059958 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.059885 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:20.060184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060172 5577 cpu_manager.go:414] "containerMap: removing container" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="prom-label-proxy" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060186 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="prom-label-proxy" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060196 5577 cpu_manager.go:414] "containerMap: removing container" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="method-gate" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060201 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="method-gate" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060211 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="prom-label-proxy" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060216 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="prom-label-proxy" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060224 5577 cpu_manager.go:414] "containerMap: removing container" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="kube-rbac-proxy" Oct 31 00:50:20.060227 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060228 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="kube-rbac-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060236 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="kube-rbac-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060241 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="kube-rbac-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060247 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="config-reloader" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060251 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="config-reloader" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060258 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="init-config-reloader" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060262 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="init-config-reloader" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060280 5577 cpu_manager.go:414] "containerMap: removing container" podUID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" containerName="perses" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060284 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" containerName="perses" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060290 5577 cpu_manager.go:414] "containerMap: removing container" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="method-gate" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060296 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="method-gate" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060302 5577 cpu_manager.go:414] "containerMap: removing container" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060306 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060316 5577 cpu_manager.go:414] "containerMap: removing container" podUID="90120372-c1e7-4e29-9c9b-4591e8d48969" containerName="otc-container" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060320 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90120372-c1e7-4e29-9c9b-4591e8d48969" containerName="otc-container" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060359 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="prom-label-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060368 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="method-gate" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060374 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="prom-label-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060380 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d2643-213a-4cd6-8b2b-0719746a9131" containerName="kube-rbac-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060385 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="kube-rbac-proxy" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060389 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="config-reloader" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060396 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="060b1e0b-7dfd-4168-8368-8fdd42f28ef0" containerName="perses" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060401 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2aff232-f1a9-4382-9619-d9375b5bfac2" containerName="method-gate" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060406 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" containerName="alertmanager" Oct 31 00:50:20.060448 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.060411 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="90120372-c1e7-4e29-9c9b-4591e8d48969" containerName="otc-container" Oct 31 00:50:20.065098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.065078 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.074450 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.074423 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:20.089543 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.089504 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:20.090526 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.090475 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 31 00:50:20.090833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.090817 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-jhdvr\"" Oct 31 00:50:20.093107 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.093093 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.095692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.095667 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.104301 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.104279 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:20.106814 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.106786 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.106925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.106878 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.106925 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.106912 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4rh\" (UniqueName: \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.107043 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.106955 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.107098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.107041 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmsl2\" (UniqueName: \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.107098 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.107091 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.107238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.107217 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.107299 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.107278 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.124867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.124840 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 31 00:50:20.124867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.124867 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-p48wv\"" Oct 31 00:50:20.148841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148809 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 31 00:50:20.148841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148824 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 00:50:20.148841 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148827 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 31 00:50:20.149091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148811 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 31 00:50:20.149091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148878 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 31 00:50:20.149091 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.148883 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 31 00:50:20.208739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208678 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.208739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208746 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208778 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4rh\" (UniqueName: \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208813 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208849 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmsl2\" (UniqueName: \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208878 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208899 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.208926 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209021 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.209015 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-prometheus-namespace-proxy-tls: secret "data-science-prometheus-namespace-proxy-tls" not found Oct 31 00:50:20.209338 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.209027 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:50:20.209338 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.209084 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls podName:3087bd4b-d5ec-4425-b45e-c8751ca6ad8f nodeName:}" failed. No retries permitted until 2026-10-31 00:50:20.709062356 +0000 UTC m=+3059.700721936 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls") pod "data-science-prometheus-namespace-proxy-5775598bd8-km4gp" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f") : secret "data-science-prometheus-namespace-proxy-tls" not found Oct 31 00:50:20.209338 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.209105 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls podName:74ef4dce-20b7-4e87-80aa-b34618de5cc8 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:20.709096215 +0000 UTC m=+3059.700755780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls") pod "data-science-thanos-querier-proxy-54fbc887bb-pwqt7" (UID: "74ef4dce-20b7-4e87-80aa-b34618de5cc8") : secret "data-science-thanos-querier-proxy-tls" not found Oct 31 00:50:20.209471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.209434 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209520 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.209510 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.209613 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.209587 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.209655 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.209639 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.222402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.222369 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmsl2\" (UniqueName: \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.222402 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.222372 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4rh\" (UniqueName: \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.223452 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.223421 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.356484 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.356391 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.494798 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.494762 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.592155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.592123 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:20.596994 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.596965 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.607784 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.607479 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:20.611785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.611758 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.611785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.611795 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.612156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.611906 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75rw7\" (UniqueName: \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.612156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.611993 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.612156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.612032 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.612156 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.612118 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.612383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.612350 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 31 00:50:20.627020 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.626990 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:20.630439 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.630408 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.641468 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.641440 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:20.678281 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.678245 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.691474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691411 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-perses-config\"" Oct 31 00:50:20.691474 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691411 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 31 00:50:20.691747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691598 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-gwktf\"" Oct 31 00:50:20.691747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691612 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 31 00:50:20.691747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691664 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:50:20.691919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691770 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 31 00:50:20.691919 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.691826 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 31 00:50:20.712781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712747 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.712781 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712783 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.712996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712812 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.712996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712853 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75rw7\" (UniqueName: \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.712996 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712903 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.713130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.712992 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.713130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.713034 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.713130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.713062 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.713130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.713098 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.713130 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.713125 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.713411 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.713179 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqb65\" (UniqueName: \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.713411 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.713238 5577 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 31 00:50:20.713411 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:20.713308 5577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs podName:9a0c2868-149f-4615-8721-ce21472f0686 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:21.213287379 +0000 UTC m=+3060.204946963 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "9a0c2868-149f-4615-8721-ce21472f0686") : secret "data-science-collector-collector-headless-tls" not found Oct 31 00:50:20.714121 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.714094 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.715633 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.715609 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") pod \"data-science-thanos-querier-proxy-54fbc887bb-pwqt7\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:20.715786 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.715621 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") pod \"data-science-prometheus-namespace-proxy-5775598bd8-km4gp\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:20.715867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.715832 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.716018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.715999 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.716582 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.716561 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.731765 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.731738 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75rw7\" (UniqueName: \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:20.805724 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.805672 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:20.814532 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.814502 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.814660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.814560 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqb65\" (UniqueName: \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.814660 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.814590 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.814986 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.814970 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.815164 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.815134 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.827323 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.827300 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqb65\" (UniqueName: \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:20.983828 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:20.983733 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:21.011412 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.011373 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:21.068241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.068161 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:21.129876 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.129841 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5edc87-63a3-4560-8df9-a1030334c875" path="/var/lib/kubelet/pods/4a5edc87-63a3-4560-8df9-a1030334c875/volumes" Oct 31 00:50:21.217804 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.217772 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:21.220692 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.220665 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:21.269857 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.269825 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:21.273637 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:50:21.273608 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3087bd4b_d5ec_4425_b45e_c8751ca6ad8f.slice/crio-60d42b735feb008b71e20eb8c73fd983bbd464220ecefccdf196785b3b143340 WatchSource:0}: Error finding container 60d42b735feb008b71e20eb8c73fd983bbd464220ecefccdf196785b3b143340: Status 404 returned error can't find the container with id 60d42b735feb008b71e20eb8c73fd983bbd464220ecefccdf196785b3b143340 Oct 31 00:50:21.317031 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.317003 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:21.322550 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:50:21.322521 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ef4dce_20b7_4e87_80aa_b34618de5cc8.slice/crio-5a28f371ab1826e3ca0cd4d3af3ecf633f7d6ac580b052b6af3272f695f7765e WatchSource:0}: Error finding container 5a28f371ab1826e3ca0cd4d3af3ecf633f7d6ac580b052b6af3272f695f7765e: Status 404 returned error can't find the container with id 5a28f371ab1826e3ca0cd4d3af3ecf633f7d6ac580b052b6af3272f695f7765e Oct 31 00:50:21.462725 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.462680 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:21.513062 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.513037 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-gwktf\"" Oct 31 00:50:21.521289 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.521228 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:21.676600 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.676557 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:21.852080 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.852011 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:21.854998 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:50:21.854974 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0c2868_149f_4615_8721_ce21472f0686.slice/crio-2d314da2da7093629042f027dd66c8f7979052c16d6deef1c2a9df8e84485e2a WatchSource:0}: Error finding container 2d314da2da7093629042f027dd66c8f7979052c16d6deef1c2a9df8e84485e2a: Status 404 returned error can't find the container with id 2d314da2da7093629042f027dd66c8f7979052c16d6deef1c2a9df8e84485e2a Oct 31 00:50:21.857340 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.857306 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:21.857455 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.857373 5577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Oct 31 00:50:21.885326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.884761 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerStarted","Data":"7a5d62c863dc3d95f51b2e4810586d0c5a658604f7300a291a046beb28bb1318"} Oct 31 00:50:21.885326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.884853 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerStarted","Data":"c4112cae2415b4947ce288851d29bc9dde44e9a75c9c259baaf4affb0c5a0981"} Oct 31 00:50:21.885326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.884876 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerStarted","Data":"e861b17c67cf5907ae109ae6f1e4af7b56a19448defa9ceafea50414c358a610"} Oct 31 00:50:21.885326 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.884898 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerStarted","Data":"60d42b735feb008b71e20eb8c73fd983bbd464220ecefccdf196785b3b143340"} Oct 31 00:50:21.886035 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.886009 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9a0c2868-149f-4615-8721-ce21472f0686","Type":"ContainerStarted","Data":"2d314da2da7093629042f027dd66c8f7979052c16d6deef1c2a9df8e84485e2a"} Oct 31 00:50:21.889756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.888225 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerStarted","Data":"762b5405af3fb305a5022071f2d4514a6139a473b33960ae917dd628e6e17653"} Oct 31 00:50:21.889756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.888267 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerStarted","Data":"1dec65d4f2478e9f797e686344d98ddc6f32508df906e837a9bfe43153abbe61"} Oct 31 00:50:21.889756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.888290 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerStarted","Data":"ee94b6bd1ccecf49e3cf7895343c9ae7bcb654065be5dba7696e0a7c2d6650cd"} Oct 31 00:50:21.889756 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.888309 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerStarted","Data":"5a28f371ab1826e3ca0cd4d3af3ecf633f7d6ac580b052b6af3272f695f7765e"} Oct 31 00:50:21.907835 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:21.907635 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" podStartSLOduration=1.907621157 podStartE2EDuration="1.907621157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:21.906207752 +0000 UTC m=+3060.897867337" watchObservedRunningTime="2026-10-31 00:50:21.907621157 +0000 UTC m=+3060.899280742" Oct 31 00:50:22.894548 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:22.894468 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9a0c2868-149f-4615-8721-ce21472f0686","Type":"ContainerStarted","Data":"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46"} Oct 31 00:50:22.916949 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:22.916902 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" podStartSLOduration=2.916889644 podStartE2EDuration="2.916889644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:21.929075702 +0000 UTC m=+3060.920735298" watchObservedRunningTime="2026-10-31 00:50:22.916889644 +0000 UTC m=+3061.908549227" Oct 31 00:50:22.917238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:22.917212 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.917203042 podStartE2EDuration="2.917203042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:22.916577284 +0000 UTC m=+3061.908236870" watchObservedRunningTime="2026-10-31 00:50:22.917203042 +0000 UTC m=+3061.908862626" Oct 31 00:50:23.271259 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:23.271218 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:25.654037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.654001 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.661052 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.661029 5577 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") device path: \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.755255 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.755229 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.755416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.755324 5577 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.758686 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.758666 5577 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.856381 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.856352 5577 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/550f1e7a650c9c0a0c356e3386fe4fd3d74af697c91f04a25a6c2eb10b2bed68/globalmount\"" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:25.874697 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:25.874674 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"data-science-perses-0\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:26.004527 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.004493 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.045445 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.045424 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"default-dockercfg-2fn9l\"" Oct 31 00:50:26.052801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.052775 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:26.220904 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.220869 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.326463 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.326369 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:26.328839 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:50:26.328813 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode223965f_a3d9_4daf_98bb_a08a352fb1b3.slice/crio-881b9e2b26ee8fa6bf90994b119baaf6e7d667835f34c4559aaaf1c9b68a4d9a WatchSource:0}: Error finding container 881b9e2b26ee8fa6bf90994b119baaf6e7d667835f34c4559aaaf1c9b68a4d9a: Status 404 returned error can't find the container with id 881b9e2b26ee8fa6bf90994b119baaf6e7d667835f34c4559aaaf1c9b68a4d9a Oct 31 00:50:26.355486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.355448 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.541615 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.541574 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:26.547743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.547690 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.552241 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.552209 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 31 00:50:26.552371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.552321 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 31 00:50:26.552371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.552350 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 31 00:50:26.552471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.552396 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 31 00:50:26.552471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.552396 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-rdq4v\"" Oct 31 00:50:26.559951 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.559921 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:26.562822 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.562789 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cccs\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.562952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.562849 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.562952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.562909 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.562952 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.562932 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.563128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.562995 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.563128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.563052 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.563128 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.563077 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.592506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.592431 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.664303 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664270 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664307 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664332 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cccs\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664365 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664421 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664448 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.664799 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.664507 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.667682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.667616 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.667837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.667735 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.667837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.667752 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.667837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.667780 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.668055 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.668035 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.668454 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.668430 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.678581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.678551 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cccs\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.781581 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.781524 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:26.868558 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.868477 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:26.917890 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.917853 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"e223965f-a3d9-4daf-98bb-a08a352fb1b3","Type":"ContainerStarted","Data":"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61"} Oct 31 00:50:26.918061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.917911 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:26.918061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.917926 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"e223965f-a3d9-4daf-98bb-a08a352fb1b3","Type":"ContainerStarted","Data":"881b9e2b26ee8fa6bf90994b119baaf6e7d667835f34c4559aaaf1c9b68a4d9a"} Oct 31 00:50:26.939074 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:26.939001 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-perses-0" podStartSLOduration=6.938981276 podStartE2EDuration="6.938981276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:26.93884369 +0000 UTC m=+3065.930503293" watchObservedRunningTime="2026-10-31 00:50:26.938981276 +0000 UTC m=+3065.930640861" Oct 31 00:50:27.171276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:27.171198 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:27.246647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:27.246588 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:27.924360 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:27.924319 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerStarted","Data":"b5627877186323b005c65987ab0daba136b12df41680a66102df5f88ea805cc8"} Oct 31 00:50:27.924853 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:27.924386 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerStarted","Data":"3c304b810862f15410db9bb2b516e4be2947ff7eac0ecc962c66dc2c3ed4dc1b"} Oct 31 00:50:29.291592 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:29.291515 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:50:31.944309 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:31.944275 5577 generic.go:356] "Generic (PLEG): container finished" podID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerID="b5627877186323b005c65987ab0daba136b12df41680a66102df5f88ea805cc8" exitCode=0 Oct 31 00:50:31.944723 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:31.944344 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerDied","Data":"b5627877186323b005c65987ab0daba136b12df41680a66102df5f88ea805cc8"} Oct 31 00:50:32.950459 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:32.950421 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerStarted","Data":"3e9c9d4be7dee0944c5bd3399335a0ee77ce1437269b4911e5eca7dccef3c56b"} Oct 31 00:50:32.950893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:32.950510 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerStarted","Data":"4a3f4065ac6119b87bf400469afdbe3c8579cfd8d9ad2b06180a991f0690cd41"} Oct 31 00:50:32.950893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:32.950697 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:32.979454 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:32.979393 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.3948514 podStartE2EDuration="6.979379436s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.584528036s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:32.977647585 +0000 UTC m=+3071.969307182" watchObservedRunningTime="2026-10-31 00:50:32.979379436 +0000 UTC m=+3071.971039020" Oct 31 00:50:37.926754 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:37.926698 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:38.955983 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:38.955950 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:50:53.721042 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.720956 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:53.721670 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.721395 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="kube-rbac-proxy" containerID="cri-o://e861b17c67cf5907ae109ae6f1e4af7b56a19448defa9ceafea50414c358a610" gracePeriod=30 Oct 31 00:50:53.721747 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.721692 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="prom-label-proxy" containerID="cri-o://7a5d62c863dc3d95f51b2e4810586d0c5a658604f7300a291a046beb28bb1318" gracePeriod=30 Oct 31 00:50:53.721826 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.721810 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="method-gate" containerID="cri-o://c4112cae2415b4947ce288851d29bc9dde44e9a75c9c259baaf4affb0c5a0981" gracePeriod=30 Oct 31 00:50:53.722724 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.722393 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:53.722724 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.722625 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="kube-rbac-proxy" containerID="cri-o://ee94b6bd1ccecf49e3cf7895343c9ae7bcb654065be5dba7696e0a7c2d6650cd" gracePeriod=30 Oct 31 00:50:53.722829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.722661 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="method-gate" containerID="cri-o://1dec65d4f2478e9f797e686344d98ddc6f32508df906e837a9bfe43153abbe61" gracePeriod=30 Oct 31 00:50:53.722829 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.722647 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="prom-label-proxy" containerID="cri-o://762b5405af3fb305a5022071f2d4514a6139a473b33960ae917dd628e6e17653" gracePeriod=30 Oct 31 00:50:53.929217 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.929184 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:53.939295 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.939264 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:50:53.939611 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.939579 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" containerID="cri-o://4a3f4065ac6119b87bf400469afdbe3c8579cfd8d9ad2b06180a991f0690cd41" gracePeriod=120 Oct 31 00:50:53.939785 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.939580 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="config-reloader" containerID="cri-o://3e9c9d4be7dee0944c5bd3399335a0ee77ce1437269b4911e5eca7dccef3c56b" gracePeriod=120 Oct 31 00:50:53.955770 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:53.955738 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.155:9093/-/ready\": dial tcp 10.133.0.155:9093: connect: connection refused" Oct 31 00:50:54.041183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041155 5577 generic.go:356] "Generic (PLEG): container finished" podID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerID="762b5405af3fb305a5022071f2d4514a6139a473b33960ae917dd628e6e17653" exitCode=0 Oct 31 00:50:54.041183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041178 5577 generic.go:356] "Generic (PLEG): container finished" podID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerID="1dec65d4f2478e9f797e686344d98ddc6f32508df906e837a9bfe43153abbe61" exitCode=0 Oct 31 00:50:54.041183 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041185 5577 generic.go:356] "Generic (PLEG): container finished" podID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerID="ee94b6bd1ccecf49e3cf7895343c9ae7bcb654065be5dba7696e0a7c2d6650cd" exitCode=0 Oct 31 00:50:54.041471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041230 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerDied","Data":"762b5405af3fb305a5022071f2d4514a6139a473b33960ae917dd628e6e17653"} Oct 31 00:50:54.041471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041301 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerDied","Data":"1dec65d4f2478e9f797e686344d98ddc6f32508df906e837a9bfe43153abbe61"} Oct 31 00:50:54.041471 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.041334 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerDied","Data":"ee94b6bd1ccecf49e3cf7895343c9ae7bcb654065be5dba7696e0a7c2d6650cd"} Oct 31 00:50:54.043495 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043468 5577 generic.go:356] "Generic (PLEG): container finished" podID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerID="7a5d62c863dc3d95f51b2e4810586d0c5a658604f7300a291a046beb28bb1318" exitCode=0 Oct 31 00:50:54.043607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043498 5577 generic.go:356] "Generic (PLEG): container finished" podID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerID="c4112cae2415b4947ce288851d29bc9dde44e9a75c9c259baaf4affb0c5a0981" exitCode=0 Oct 31 00:50:54.043607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043510 5577 generic.go:356] "Generic (PLEG): container finished" podID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerID="e861b17c67cf5907ae109ae6f1e4af7b56a19448defa9ceafea50414c358a610" exitCode=0 Oct 31 00:50:54.043607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043533 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerDied","Data":"7a5d62c863dc3d95f51b2e4810586d0c5a658604f7300a291a046beb28bb1318"} Oct 31 00:50:54.043607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043574 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerDied","Data":"c4112cae2415b4947ce288851d29bc9dde44e9a75c9c259baaf4affb0c5a0981"} Oct 31 00:50:54.043607 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.043598 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerDied","Data":"e861b17c67cf5907ae109ae6f1e4af7b56a19448defa9ceafea50414c358a610"} Oct 31 00:50:54.051924 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.051901 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:54.054905 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.054886 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:54.092022 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.091992 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\" (UniqueName: \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\") pod \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " Oct 31 00:50:54.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092047 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") pod \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " Oct 31 00:50:54.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092076 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") pod \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " Oct 31 00:50:54.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092115 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\" (UniqueName: \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\") pod \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " Oct 31 00:50:54.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092147 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\") pod \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " Oct 31 00:50:54.092184 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092172 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\" (UniqueName: \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\") pod \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\" (UID: \"74ef4dce-20b7-4e87-80aa-b34618de5cc8\") " Oct 31 00:50:54.092466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092205 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\") pod \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " Oct 31 00:50:54.092466 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092241 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\") pod \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\" (UID: \"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f\") " Oct 31 00:50:54.092794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092770 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca" pod "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.092794 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.092769 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config" pod "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.093185 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.093160 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config" pod "74ef4dce-20b7-4e87-80aa-b34618de5cc8" (UID: "74ef4dce-20b7-4e87-80aa-b34618de5cc8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.093426 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.093238 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca" pod "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.096524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.096501 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:54.096737 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.096685 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9a0c2868-149f-4615-8721-ce21472f0686" containerName="otc-container" containerID="cri-o://e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46" gracePeriod=30 Oct 31 00:50:54.098129 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.098098 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh" pod "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f"). InnerVolumeSpecName "kube-api-access-xp4rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.098727 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.098688 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls" pod "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" (UID: "3087bd4b-d5ec-4425-b45e-c8751ca6ad8f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.099420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.098785 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls" pod "74ef4dce-20b7-4e87-80aa-b34618de5cc8" (UID: "74ef4dce-20b7-4e87-80aa-b34618de5cc8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.099420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.099284 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2" pod "74ef4dce-20b7-4e87-80aa-b34618de5cc8" (UID: "74ef4dce-20b7-4e87-80aa-b34618de5cc8"). InnerVolumeSpecName "kube-api-access-zmsl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.181203 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.181170 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:54.181420 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.181345 5577 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-perses-0" podUID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" containerName="perses" containerID="cri-o://bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61" gracePeriod=30 Oct 31 00:50:54.193298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193273 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193297 5577 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-client-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193306 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmsl2\" (UniqueName: \"kubernetes.io/projected/74ef4dce-20b7-4e87-80aa-b34618de5cc8-kube-api-access-zmsl2\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193316 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193324 5577 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/74ef4dce-20b7-4e87-80aa-b34618de5cc8-tls\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193332 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4rh\" (UniqueName: \"kubernetes.io/projected/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-kube-api-access-xp4rh\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193339 5577 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f-prometheus-tls-ca\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.193510 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.193353 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/74ef4dce-20b7-4e87-80aa-b34618de5cc8-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.500093 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.499751 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:54.524115 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.524088 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:54.597337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597247 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597337 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597283 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597404 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") pod \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " Oct 31 00:50:54.597549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597454 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\" (UniqueName: \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\") pod \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " Oct 31 00:50:54.597549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597483 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597549 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597533 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597563 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\" (UniqueName: \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597590 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\") pod \"9a0c2868-149f-4615-8721-ce21472f0686\" (UID: \"9a0c2868-149f-4615-8721-ce21472f0686\") " Oct 31 00:50:54.597817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597620 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\" (UniqueName: \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\") pod \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " Oct 31 00:50:54.597817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597664 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\" (UniqueName: \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\") pod \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\" (UID: \"e223965f-a3d9-4daf-98bb-a08a352fb1b3\") " Oct 31 00:50:54.597817 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597762 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.598079 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.597930 5577 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9a0c2868-149f-4615-8721-ce21472f0686-otc-internal\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.598739 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.598683 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config" pod "e223965f-a3d9-4daf-98bb-a08a352fb1b3" (UID: "e223965f-a3d9-4daf-98bb-a08a352fb1b3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:54.600230 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.600190 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.600424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.600300 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.600424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.600360 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.600583 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.600546 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65" pod "e223965f-a3d9-4daf-98bb-a08a352fb1b3" (UID: "e223965f-a3d9-4daf-98bb-a08a352fb1b3"). InnerVolumeSpecName "kube-api-access-mqb65". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.601385 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.601361 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6" pod "e223965f-a3d9-4daf-98bb-a08a352fb1b3" (UID: "e223965f-a3d9-4daf-98bb-a08a352fb1b3"). InnerVolumeSpecName "pvc-6345ce72-7a68-409a-80dd-f0274e634770". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 31 00:50:54.601486 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.601366 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:54.601762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.601681 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7" pod "9a0c2868-149f-4615-8721-ce21472f0686" (UID: "9a0c2868-149f-4615-8721-ce21472f0686"). InnerVolumeSpecName "kube-api-access-75rw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:54.698837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698803 5577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.698837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698834 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75rw7\" (UniqueName: \"kubernetes.io/projected/9a0c2868-149f-4615-8721-ce21472f0686-kube-api-access-75rw7\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.698837 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698846 5577 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.699082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698855 5577 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/e223965f-a3d9-4daf-98bb-a08a352fb1b3-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.699082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698864 5577 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-otlp-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.699082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698892 5577 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" " Oct 31 00:50:54.699082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698901 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqb65\" (UniqueName: \"kubernetes.io/projected/e223965f-a3d9-4daf-98bb-a08a352fb1b3-kube-api-access-mqb65\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.699082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.698910 5577 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9a0c2868-149f-4615-8721-ce21472f0686-monitoring-tls-certs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.711762 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.711742 5577 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-6345ce72-7a68-409a-80dd-f0274e634770" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6") on node "ip-10-0-136-236.us-east-2.compute.internal" Oct 31 00:50:54.750378 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.750344 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins" pod "e223965f-a3d9-4daf-98bb-a08a352fb1b3" (UID: "e223965f-a3d9-4daf-98bb-a08a352fb1b3"). InnerVolumeSpecName "plugins". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:54.800018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.799985 5577 reconciler_common.go:299] "Volume detached for volume \"plugins\" (UniqueName: \"kubernetes.io/empty-dir/e223965f-a3d9-4daf-98bb-a08a352fb1b3-plugins\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:54.800018 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:54.800013 5577 reconciler_common.go:299] "Volume detached for volume \"pvc-6345ce72-7a68-409a-80dd-f0274e634770\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-00dc4657431fd3bd6\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"csi-7dc54017043cacf3b82ddb16a4696edffbe5be40642f1fabaa24a6c82d704ed3\"" Oct 31 00:50:55.048506 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.048473 5577 generic.go:356] "Generic (PLEG): container finished" podID="9a0c2868-149f-4615-8721-ce21472f0686" containerID="e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46" exitCode=0 Oct 31 00:50:55.048682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.048520 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 31 00:50:55.048682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.048549 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9a0c2868-149f-4615-8721-ce21472f0686","Type":"ContainerDied","Data":"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46"} Oct 31 00:50:55.048682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.048599 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9a0c2868-149f-4615-8721-ce21472f0686","Type":"ContainerDied","Data":"2d314da2da7093629042f027dd66c8f7979052c16d6deef1c2a9df8e84485e2a"} Oct 31 00:50:55.048682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.048624 5577 scope.go:122] "RemoveContainer" containerID="e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46" Oct 31 00:50:55.050873 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.050848 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" Oct 31 00:50:55.050972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.050861 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7" event={"ID":"74ef4dce-20b7-4e87-80aa-b34618de5cc8","Type":"ContainerDied","Data":"5a28f371ab1826e3ca0cd4d3af3ecf633f7d6ac580b052b6af3272f695f7765e"} Oct 31 00:50:55.053320 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.053300 5577 generic.go:356] "Generic (PLEG): container finished" podID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerID="3e9c9d4be7dee0944c5bd3399335a0ee77ce1437269b4911e5eca7dccef3c56b" exitCode=0 Oct 31 00:50:55.053407 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.053379 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerDied","Data":"3e9c9d4be7dee0944c5bd3399335a0ee77ce1437269b4911e5eca7dccef3c56b"} Oct 31 00:50:55.054635 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.054610 5577 generic.go:356] "Generic (PLEG): container finished" podID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" containerID="bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61" exitCode=0 Oct 31 00:50:55.054761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.054653 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-perses-0" Oct 31 00:50:55.054761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.054691 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"e223965f-a3d9-4daf-98bb-a08a352fb1b3","Type":"ContainerDied","Data":"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61"} Oct 31 00:50:55.054761 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.054741 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-perses-0" event={"ID":"e223965f-a3d9-4daf-98bb-a08a352fb1b3","Type":"ContainerDied","Data":"881b9e2b26ee8fa6bf90994b119baaf6e7d667835f34c4559aaaf1c9b68a4d9a"} Oct 31 00:50:55.056774 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.056755 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" event={"ID":"3087bd4b-d5ec-4425-b45e-c8751ca6ad8f","Type":"ContainerDied","Data":"60d42b735feb008b71e20eb8c73fd983bbd464220ecefccdf196785b3b143340"} Oct 31 00:50:55.056849 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.056814 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp" Oct 31 00:50:55.059213 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.059197 5577 scope.go:122] "RemoveContainer" containerID="e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46" Oct 31 00:50:55.059509 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:55.059483 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46\": container with ID starting with e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46 not found: ID does not exist" containerID="e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46" Oct 31 00:50:55.059610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.059519 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46"} err="failed to get container status \"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46\": rpc error: code = NotFound desc = could not find container \"e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46\": container with ID starting with e42042d91707e973441529e3f76c76d793ab2d13bea4df6f7a7e2e44d2f26b46 not found: ID does not exist" Oct 31 00:50:55.059610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.059540 5577 scope.go:122] "RemoveContainer" containerID="762b5405af3fb305a5022071f2d4514a6139a473b33960ae917dd628e6e17653" Oct 31 00:50:55.069026 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.069010 5577 scope.go:122] "RemoveContainer" containerID="1dec65d4f2478e9f797e686344d98ddc6f32508df906e837a9bfe43153abbe61" Oct 31 00:50:55.076423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.076408 5577 scope.go:122] "RemoveContainer" containerID="ee94b6bd1ccecf49e3cf7895343c9ae7bcb654065be5dba7696e0a7c2d6650cd" Oct 31 00:50:55.083797 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.083777 5577 scope.go:122] "RemoveContainer" containerID="bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61" Oct 31 00:50:55.087377 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.087355 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:55.091593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.091576 5577 scope.go:122] "RemoveContainer" containerID="bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61" Oct 31 00:50:55.092115 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:55.092085 5577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61\": container with ID starting with bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61 not found: ID does not exist" containerID="bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61" Oct 31 00:50:55.092278 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.092252 5577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61"} err="failed to get container status \"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61\": rpc error: code = NotFound desc = could not find container \"bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61\": container with ID starting with bad8d929191264e8930b5ccafe9da4814db30d2aaf0b4df69eb6b8ba2e5d4c61 not found: ID does not exist" Oct 31 00:50:55.092397 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.092377 5577 scope.go:122] "RemoveContainer" containerID="7a5d62c863dc3d95f51b2e4810586d0c5a658604f7300a291a046beb28bb1318" Oct 31 00:50:55.093593 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.093575 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-54fbc887bb-pwqt7"] Oct 31 00:50:55.100610 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.100594 5577 scope.go:122] "RemoveContainer" containerID="c4112cae2415b4947ce288851d29bc9dde44e9a75c9c259baaf4affb0c5a0981" Oct 31 00:50:55.105880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.105860 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:55.108438 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.108424 5577 scope.go:122] "RemoveContainer" containerID="e861b17c67cf5907ae109ae6f1e4af7b56a19448defa9ceafea50414c358a610" Oct 31 00:50:55.110902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.110881 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 31 00:50:55.130127 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.130102 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" path="/var/lib/kubelet/pods/74ef4dce-20b7-4e87-80aa-b34618de5cc8/volumes" Oct 31 00:50:55.130567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.130552 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a0c2868-149f-4615-8721-ce21472f0686" path="/var/lib/kubelet/pods/9a0c2868-149f-4615-8721-ce21472f0686/volumes" Oct 31 00:50:55.130801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.130788 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:55.130846 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.130817 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-5775598bd8-km4gp"] Oct 31 00:50:55.143741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.143700 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:55.150464 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:55.150444 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-perses-0"] Oct 31 00:50:56.032684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:56.032651 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:50:57.129263 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:57.129231 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" path="/var/lib/kubelet/pods/3087bd4b-d5ec-4425-b45e-c8751ca6ad8f/volumes" Oct 31 00:50:57.129741 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:57.129701 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" path="/var/lib/kubelet/pods/e223965f-a3d9-4daf-98bb-a08a352fb1b3/volumes" Oct 31 00:50:58.955743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:50:58.955683 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.155:9093/-/ready\": dial tcp 10.133.0.155:9093: connect: connection refused" Oct 31 00:50:59.291352 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:50:59.291315 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:51:03.955704 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:03.955671 5577 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" probeResult="failure" output="Get \"http://10.133.0.155:9093/-/ready\": dial tcp 10.133.0.155:9093: connect: connection refused" Oct 31 00:51:04.105547 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.105513 5577 generic.go:356] "Generic (PLEG): container finished" podID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerID="4a3f4065ac6119b87bf400469afdbe3c8579cfd8d9ad2b06180a991f0690cd41" exitCode=0 Oct 31 00:51:04.105722 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.105586 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerDied","Data":"4a3f4065ac6119b87bf400469afdbe3c8579cfd8d9ad2b06180a991f0690cd41"} Oct 31 00:51:04.187191 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.187168 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:51:04.278075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278041 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278082 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278276 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278117 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278291 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278349 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278416 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278402 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278515 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278449 5577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\") pod \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\" (UID: \"8177e670-1872-4e9d-8a6f-94cd0ef6a293\") " Oct 31 00:51:04.278902 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.278836 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:04.280668 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.280642 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:51:04.280903 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.280881 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:04.280955 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.280911 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "kube-api-access-8cccs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:51:04.281976 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.281955 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:04.285152 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.285126 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:04.292168 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.292145 5577 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config" pod "8177e670-1872-4e9d-8a6f-94cd0ef6a293" (UID: "8177e670-1872-4e9d-8a6f-94cd0ef6a293"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:51:04.379989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.379949 5577 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-web-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.379989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.379974 5577 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-cluster-tls-config\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.379989 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.379993 5577 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-volume\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.380238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.380002 5577 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.380238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.380013 5577 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/8177e670-1872-4e9d-8a6f-94cd0ef6a293-config-out\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.380238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.380021 5577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cccs\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-kube-api-access-8cccs\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:04.380238 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:04.380030 5577 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/8177e670-1872-4e9d-8a6f-94cd0ef6a293-tls-assets\") on node \"ip-10-0-136-236.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:51:05.111488 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.111454 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"8177e670-1872-4e9d-8a6f-94cd0ef6a293","Type":"ContainerDied","Data":"3c304b810862f15410db9bb2b516e4be2947ff7eac0ecc962c66dc2c3ed4dc1b"} Oct 31 00:51:05.111954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.111523 5577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 31 00:51:05.111954 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.111563 5577 scope.go:122] "RemoveContainer" containerID="3e9c9d4be7dee0944c5bd3399335a0ee77ce1437269b4911e5eca7dccef3c56b" Oct 31 00:51:05.120198 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.120177 5577 scope.go:122] "RemoveContainer" containerID="4a3f4065ac6119b87bf400469afdbe3c8579cfd8d9ad2b06180a991f0690cd41" Oct 31 00:51:05.127971 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.127954 5577 scope.go:122] "RemoveContainer" containerID="b5627877186323b005c65987ab0daba136b12df41680a66102df5f88ea805cc8" Oct 31 00:51:05.135383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.135358 5577 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:51:05.139783 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:05.139759 5577 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 31 00:51:07.128045 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:07.128012 5577 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" path="/var/lib/kubelet/pods/8177e670-1872-4e9d-8a6f-94cd0ef6a293/volumes" Oct 31 00:51:10.266451 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:10.266422 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:15.995567 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:15.995528 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:16.134332 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:16.134299 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:20.740357 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:20.740318 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-observability-system_odh-observability-85cd6b576d-z5f58_c2d7942b-59c9-4d34-bf75-4156c16e5bbd/manager/2.log" Oct 31 00:51:21.148061 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148029 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtkzz/must-gather-clt64"] Oct 31 00:51:21.148298 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148287 5577 cpu_manager.go:414] "containerMap: removing container" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="prom-label-proxy" Oct 31 00:51:21.148343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148299 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="prom-label-proxy" Oct 31 00:51:21.148343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148316 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="config-reloader" Oct 31 00:51:21.148343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148322 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="config-reloader" Oct 31 00:51:21.148343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148332 5577 cpu_manager.go:414] "containerMap: removing container" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="method-gate" Oct 31 00:51:21.148343 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148337 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="method-gate" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148346 5577 cpu_manager.go:414] "containerMap: removing container" podUID="9a0c2868-149f-4615-8721-ce21472f0686" containerName="otc-container" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148351 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a0c2868-149f-4615-8721-ce21472f0686" containerName="otc-container" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148358 5577 cpu_manager.go:414] "containerMap: removing container" podUID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" containerName="perses" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148362 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" containerName="perses" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148377 5577 cpu_manager.go:414] "containerMap: removing container" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="method-gate" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148382 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="method-gate" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148387 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148392 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148399 5577 cpu_manager.go:414] "containerMap: removing container" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="kube-rbac-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148403 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="kube-rbac-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148410 5577 cpu_manager.go:414] "containerMap: removing container" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="prom-label-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148414 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="prom-label-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148420 5577 cpu_manager.go:414] "containerMap: removing container" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="init-config-reloader" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148424 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="init-config-reloader" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148430 5577 cpu_manager.go:414] "containerMap: removing container" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="kube-rbac-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148434 5577 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="kube-rbac-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148475 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="kube-rbac-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148483 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="prom-label-proxy" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148490 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="config-reloader" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148495 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="method-gate" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148501 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a0c2868-149f-4615-8721-ce21472f0686" containerName="otc-container" Oct 31 00:51:21.148500 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148505 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="e223965f-a3d9-4daf-98bb-a08a352fb1b3" containerName="perses" Oct 31 00:51:21.149223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148512 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="method-gate" Oct 31 00:51:21.149223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148517 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="8177e670-1872-4e9d-8a6f-94cd0ef6a293" containerName="alertmanager" Oct 31 00:51:21.149223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148522 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="3087bd4b-d5ec-4425-b45e-c8751ca6ad8f" containerName="kube-rbac-proxy" Oct 31 00:51:21.149223 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.148528 5577 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ef4dce-20b7-4e87-80aa-b34618de5cc8" containerName="prom-label-proxy" Oct 31 00:51:21.151354 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.151337 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.155269 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.155242 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"kube-root-ca.crt\"" Oct 31 00:51:21.155874 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.155854 5577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtkzz\"/\"openshift-service-ca.crt\"" Oct 31 00:51:21.155972 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.155957 5577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xtkzz\"/\"default-dockercfg-bmqgh\"" Oct 31 00:51:21.159460 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.159431 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/must-gather-clt64"] Oct 31 00:51:21.181627 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.181597 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.311119 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.311087 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrdk\" (UniqueName: \"kubernetes.io/projected/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-kube-api-access-rkrdk\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.311280 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.311131 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-must-gather-output\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.340541 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.340516 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:21.412075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.412000 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrdk\" (UniqueName: \"kubernetes.io/projected/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-kube-api-access-rkrdk\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.412075 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.412041 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-must-gather-output\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.412345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.412329 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-must-gather-output\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.424507 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.424486 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrdk\" (UniqueName: \"kubernetes.io/projected/8d481bea-fdeb-4e8c-8fb0-6f911859afc2-kube-api-access-rkrdk\") pod \"must-gather-clt64\" (UID: \"8d481bea-fdeb-4e8c-8fb0-6f911859afc2\") " pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.465531 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.465508 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/must-gather-clt64" Oct 31 00:51:21.736937 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.736862 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/must-gather-clt64"] Oct 31 00:51:21.739753 ip-10-0-136-236 kubenswrapper[5577]: W1031 00:51:21.739725 5577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d481bea_fdeb_4e8c_8fb0_6f911859afc2.slice/crio-518c31ecf48b2ce0bd4b3cd5257aa2b434f6195957d74eba6693a5fa5ff7c001 WatchSource:0}: Error finding container 518c31ecf48b2ce0bd4b3cd5257aa2b434f6195957d74eba6693a5fa5ff7c001: Status 404 returned error can't find the container with id 518c31ecf48b2ce0bd4b3cd5257aa2b434f6195957d74eba6693a5fa5ff7c001 Oct 31 00:51:21.741746 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:21.741697 5577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:51:22.179057 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:22.179019 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/must-gather-clt64" event={"ID":"8d481bea-fdeb-4e8c-8fb0-6f911859afc2","Type":"ContainerStarted","Data":"518c31ecf48b2ce0bd4b3cd5257aa2b434f6195957d74eba6693a5fa5ff7c001"} Oct 31 00:51:23.187743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:23.187257 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/must-gather-clt64" event={"ID":"8d481bea-fdeb-4e8c-8fb0-6f911859afc2","Type":"ContainerStarted","Data":"923b689283d9901dc6972011dbf8ff3d6025f2736e423dd090b318aaf66e3181"} Oct 31 00:51:23.187743 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:23.187309 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/must-gather-clt64" event={"ID":"8d481bea-fdeb-4e8c-8fb0-6f911859afc2","Type":"ContainerStarted","Data":"65dc39f4a4dbe297848b43055091b3bbc2cc945f8db44a56c0f4a8e2b72c8ef2"} Oct 31 00:51:23.206533 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:23.206485 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xtkzz/must-gather-clt64" podStartSLOduration=1.269501528 podStartE2EDuration="2.206470608s" totalImagesPullingTime="936.96908ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:23.206087809 +0000 UTC m=+3122.197747394" watchObservedRunningTime="2026-10-31 00:51:23.206470608 +0000 UTC m=+3122.198130192" Oct 31 00:51:23.398736 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:23.398690 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:23.538383 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:23.538323 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:24.317185 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:24.317144 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zqb2v_dfd6c668-59d3-4840-8ccc-e4e9c7066b44/global-pull-secret-syncer/0.log" Oct 31 00:51:24.498539 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:24.498507 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" Oct 31 00:51:24.693082 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:24.693005 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-236.us-east-2.compute.internal_bba88490f41dac4679774c025c13b2ce/haproxy/0.log" Oct 31 00:51:27.302424 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:27.302386 5577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-hbqzw_118f5c91-d3a9-434b-b931-89be645e00e2/konnectivity-agent/0.log" limit=1024 Oct 31 00:51:29.334397 ip-10-0-136-236 kubenswrapper[5577]: E1031 00:51:29.334242 5577 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-d8zkg" containerName="extractor" Oct 31 00:51:30.738690 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:30.738653 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-rjxm9_fe424bc5-4967-481f-bc33-3f7b3db28a9e/monitoring-plugin/0.log" Oct 31 00:51:30.888040 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:30.887982 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mddnv_9ac9fb68-1982-42a6-8a34-7b9fa61a7565/node-exporter/0.log" Oct 31 00:51:30.928682 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:30.928636 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mddnv_9ac9fb68-1982-42a6-8a34-7b9fa61a7565/kube-rbac-proxy/0.log" Oct 31 00:51:30.990535 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:30.990387 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mddnv_9ac9fb68-1982-42a6-8a34-7b9fa61a7565/init-textfile/0.log" Oct 31 00:51:31.278867 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.278814 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/prometheus/0.log" Oct 31 00:51:31.311226 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.311146 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/config-reloader/0.log" Oct 31 00:51:31.340893 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.340863 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/thanos-sidecar/0.log" Oct 31 00:51:31.379687 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.379633 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/kube-rbac-proxy-web/0.log" Oct 31 00:51:31.408181 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.408150 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/kube-rbac-proxy/0.log" Oct 31 00:51:31.443336 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.443297 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/kube-rbac-proxy-thanos/0.log" Oct 31 00:51:31.481248 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.481223 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b93e19ea-595f-458b-8919-e718a87cea5f/init-config-reloader/0.log" Oct 31 00:51:31.720537 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.718765 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/thanos-query/0.log" Oct 31 00:51:31.750914 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.750880 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/kube-rbac-proxy-web/0.log" Oct 31 00:51:31.779237 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.779208 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/kube-rbac-proxy/0.log" Oct 31 00:51:31.826371 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.826344 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/prom-label-proxy/0.log" Oct 31 00:51:31.866333 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.866306 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/kube-rbac-proxy-rules/0.log" Oct 31 00:51:31.891801 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:31.891768 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-84b5d76994-jq7jv_6aeaa875-fa12-46cc-b25d-453ce96d4a1e/kube-rbac-proxy-metrics/0.log" Oct 31 00:51:32.924908 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:32.924875 5577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg"] Oct 31 00:51:32.936943 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:32.936910 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:32.958210 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:32.947736 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg"] Oct 31 00:51:33.032399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.032082 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-lib-modules\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.032399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.032132 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwqb\" (UniqueName: \"kubernetes.io/projected/e0c05314-b91c-4e90-9171-c1b7041bba71-kube-api-access-6cwqb\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.032399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.032207 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-proc\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.032399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.032243 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-podres\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.032399 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.032273 5577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-sys\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133449 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-proc\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133524 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133511 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-podres\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133544 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-sys\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133549 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-proc\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133601 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-lib-modules\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133615 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-sys\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133618 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-podres\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133628 5577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwqb\" (UniqueName: \"kubernetes.io/projected/e0c05314-b91c-4e90-9171-c1b7041bba71-kube-api-access-6cwqb\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.133740 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.133668 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e0c05314-b91c-4e90-9171-c1b7041bba71-lib-modules\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.151380 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.151350 5577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwqb\" (UniqueName: \"kubernetes.io/projected/e0c05314-b91c-4e90-9171-c1b7041bba71-kube-api-access-6cwqb\") pod \"perf-node-gather-daemonset-m46wg\" (UID: \"e0c05314-b91c-4e90-9171-c1b7041bba71\") " pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.270037 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.270009 5577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:33.565913 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:33.565872 5577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg"] Oct 31 00:51:34.279388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:34.277923 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" event={"ID":"e0c05314-b91c-4e90-9171-c1b7041bba71","Type":"ContainerStarted","Data":"e85155ccaeda0803bdcedbc43b1d1d233967e1eb813604d599307427584d5781"} Oct 31 00:51:34.279388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:34.277988 5577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" event={"ID":"e0c05314-b91c-4e90-9171-c1b7041bba71","Type":"ContainerStarted","Data":"4f0180b73dfee50984b5ccdeb07e15d7d1b1a38ba8011738c8770195787db2b9"} Oct 31 00:51:34.279388 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:34.278766 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:34.301988 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:34.301924 5577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" podStartSLOduration=2.301904075 podStartE2EDuration="2.301904075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:34.300896567 +0000 UTC m=+3133.292556153" watchObservedRunningTime="2026-10-31 00:51:34.301904075 +0000 UTC m=+3133.293563660" Oct 31 00:51:36.377850 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:36.377822 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg8bt_692f415f-94bd-4fd2-ba6c-468145504bc0/dns/0.log" Oct 31 00:51:36.402423 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:36.402396 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gg8bt_692f415f-94bd-4fd2-ba6c-468145504bc0/kube-rbac-proxy/0.log" Oct 31 00:51:36.489347 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:36.489319 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mmztz_ccc77441-8a6d-4a2a-95ca-553f8870b21c/dns-node-resolver/0.log" Oct 31 00:51:37.104342 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:37.104314 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7k929_85bd67fa-8a46-4098-a7b2-0dab23bf0da1/node-ca/0.log" Oct 31 00:51:38.369155 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:38.369116 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8vlhd_5c12cca1-08f4-49f6-a54d-a02e676f6b54/serve-healthcheck-canary/0.log" Oct 31 00:51:38.908034 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:38.908003 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8zkg_5b456bf9-fa54-4740-998e-d0788f096c3c/kube-rbac-proxy/0.log" Oct 31 00:51:38.930860 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:38.930811 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8zkg_5b456bf9-fa54-4740-998e-d0788f096c3c/exporter/0.log" Oct 31 00:51:38.954763 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:38.954739 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-d8zkg_5b456bf9-fa54-4740-998e-d0788f096c3c/extractor/0.log" Oct 31 00:51:41.297684 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:41.297653 5577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xtkzz/perf-node-gather-daemonset-m46wg" Oct 31 00:51:41.723833 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:41.723749 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-opentelemetry-operator_opentelemetry-operator-webhook-6c7d65b66-zqwjf_259b6e3a-d9e6-4ca9-a59c-702a0bc17b98/webhook/0.log" Oct 31 00:51:42.198345 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:42.198308 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-observability-operator_obo-prometheus-operator-6ddb664c4f-bgdwj_79046fb0-7fc5-4930-8416-7d9278c90ba9/prometheus-operator/0.log" Oct 31 00:51:42.220880 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:42.220851 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-observability-operator_obo-prometheus-operator-admission-webhook-7fb7b74868-4pkk7_2cc560d9-fc04-4e45-943a-cd074028e2fb/prometheus-operator-admission-webhook/0.log" Oct 31 00:51:48.247551 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.247525 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/kube-multus-additional-cni-plugins/0.log" Oct 31 00:51:48.271023 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.270991 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/egress-router-binary-copy/0.log" Oct 31 00:51:48.294467 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.294441 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/cni-plugins/0.log" Oct 31 00:51:48.318233 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.318211 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/bond-cni-plugin/0.log" Oct 31 00:51:48.341672 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.341648 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/routeoverride-cni/0.log" Oct 31 00:51:48.365634 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.365600 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/whereabouts-cni-bincopy/0.log" Oct 31 00:51:48.398481 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.398457 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cjdd8_36bdf75e-c844-4a33-b5dd-ac33878a0186/whereabouts-cni/0.log" Oct 31 00:51:48.691334 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.691301 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lrwwf_2026287d-f8ba-4ba5-9681-d6d74a7d1058/kube-multus/0.log" Oct 31 00:51:48.799647 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.799621 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8brkm_a8f07f34-fa12-448b-8054-910fcac344a5/network-metrics-daemon/0.log" Oct 31 00:51:48.818845 ip-10-0-136-236 kubenswrapper[5577]: I1031 00:51:48.818816 5577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-8brkm_a8f07f34-fa12-448b-8054-910fcac344a5/kube-rbac-proxy/0.log"